Large language models have a peculiar problem: they will generate a response with high confidence, then reject that same response if asked to validate it. This generator-validator (G-V) gap is not a minor quirk. It undermines applications that depend on LLMs acting as their own critics, including self-refinement pipelines, chain-of-thought backtracking, and reranking approaches used in code generation and mathematical reasoning. A new paper from researchers at UT Austin, UMass Amherst, and NYU addresses this problem with a theoretically grounded training objective called FLORA (Frequency-corrected Learning of Ordered Rank Alignment). The full paper is available at arXiv:2607.02668.
The Core Problem: Why Naive Consistency Fails
The intuition behind G-V consistency seems straightforward: a model should assign high generation probability to responses it also deems valid. Prior work, particularly Rodriguez et al. (2025), has tried to enforce this by correlating generator log-odds with validator log-odds. The problem is that this framing ignores a confound that turns out to be significant in practice.
A generator assigns low probability to a string for two distinct reasons: either the string is a poor response to the prompt, or the string is simply an unlikely sequence of tokens regardless of context. A short, generic email might score poorly on a formatting instruction task not because it violates the instruction, but because it is a priori a low-frequency utterance. Conflating these two sources of low probability creates a corrupted training signal.
The authors formalise this through a rational agent model. Under this framework, a consistent agent's validator score should correlate not with the raw generator score, but with a frequency-corrected generator score: the log-probability of generating a response given a correct-answer prompt, minus the log-probability of generating that same response when an incorrect answer is requested. The latter term captures the baseline likelihood of the string, independent of correctness. The authors call this quantity the frequency-corrected generator score, and its connection to pointwise mutual information (PMI) is explicit in the paper's notation.
Methodology: FLORA as a Training Objective
FLORA operationalises this theoretical relationship as a rank-alignment loss. Rather than enforcing exact score matching, the objective encourages the validator's scores to rank responses in the same order as the frequency-corrected generator scores. This is a softer target that is more tractable for real-world LLMs, which are not literally rational agents.
Two variants of the correction are tested. The PMI correction estimates the baseline term by computing the unconditional log-probability of the response. The negative-prompting correction uses a contrastive prompt that explicitly requests an incorrect answer, taking the difference between correct-prompt and incorrect-prompt generator scores. Both are combined with standard supervised fine-tuning losses to preserve generator and validator accuracy, not just their relative ordering.
The experimental setup is worth noting for its rigour. Rather than evaluating only on responses near the mode of the model's distribution, the authors construct fixed candidate sets that include low-probability but potentially correct responses. This stress-tests the model's ability to rank correctly across the full distribution, not just at the top. Evaluation uses generator AUROC (whether the generator correctly ranks valid responses above invalid ones) and Pearson correlation between generator and validator scores. Three tasks are covered: IFEval (instruction following), HumanEval (code generation), and a hyponymy taxonomy task. Models tested include Gemma-2-9b-it, Gemma-4-31b-it, and Qwen-3.5-9B.
Results: Meaningful Gains Across Tasks
The headline numbers are strong. Key findings from Table 2 in the paper:
- FLORA-PMI improves generator AUROC by up to 7.3 percentage points over the next-best prior method across the evaluated settings.
- Generator-validator Pearson correlation improves by up to 27pp on IFEval (Gemma-2-9b-it: 62.8 vs. 35.5 for the base model) and HumanEval (FLORA-Neg: 76.3 vs. 49.0 for the next-best method).
- Validator quality is preserved or improved across all tasks, which is a critical check. Methods that improve generator scores by degrading the validator are not solving the problem.
- FLORA outperforms the prior RankAlign baseline on five of six model-task combinations on generator AUROC, and by substantial margins on G-V correlation.
The qualitative analysis in Figure 5 of the paper is particularly instructive. For a base Gemma-2-9b-it model on an IFEval prompt, the raw generator score is essentially uncorrelated with correctness (AUROC 43.1, correlation -0.24). Applying the PMI correction at test time, without any training, already recovers most of the alignment (AUROC 85.3, correlation 0.64). After FLORA-PMI training, both metrics improve further (AUROC 90.8, correlation 0.75). This decomposition suggests that a meaningful portion of the G-V gap is attributable to the frequency confound, and that the training objective is doing real work on top of the test-time correction alone.
Limitations and Open Questions
The authors are candid about where the theory-to-practice translation is imperfect. The derivation requires an assumption that a particular intractable term, denoted r in the paper, is close to 1. This is a reasonable approximation but cannot be validated directly. The empirical results support the overall approach, but the theoretical guarantees remain conditional on this assumption.
A more substantive limitation is that the evaluation framework focuses on fixed candidate sets rather than greedy decoding. The paper does not demonstrate that FLORA improves the single best output a model produces in practice. For deployment contexts where 1-best generation quality is the primary metric, this is a gap worth noting.
The binary validity assumption is also a real constraint. Many real-world tasks involve partial credit, graded correctness, or subjective quality judgements. Extending the theoretical framework to handle soft validity labels would broaden applicability considerably.
Finally, the within-prompt evaluation design, while appropriate for stress-testing distribution tails, means the results are not directly comparable to prior G-V gap measurements that aggregate across diverse prompts with sparse completions. The authors address this by re-implementing RankAlign on their own datasets, but cross-framework comparison remains indirect.
Despite these caveats, FLORA represents a principled step forward. The frequency-correction insight is theoretically well-motivated and practically effective. The connection to PMI is not novel in NLP broadly, but applying it to the G-V consistency problem is a useful contribution. For anyone building systems that rely on LLMs as self-evaluators or rerankers, this is worth reading carefully. The code and data are available at github.com/juand-r/flora.