Standard LLM benchmarks are showing their age. Datasets like MMLU and GSM8K measure correctness against fixed ground truth, which works well when answers are unambiguous. The problem is that a growing proportion of real-world LLM use cases do not have a single correct answer. Two responses to a coding question might both be technically valid, yet differ substantially in clarity, structure, and practical usefulness. Conventional benchmarks have no good way to distinguish between them.
This paper, available at arXiv:2607.21632, proposes a different approach: rather than scoring outputs against ground truth, have a panel of LLMs blindly rank each other's responses, then aggregate those rankings into a single preference metric called the Relative Intelligence Index (RII). It is a clean idea, and the paper executes it carefully enough to be worth examining in detail.
Key Contributions
The central contribution is the evaluation framework itself, which rests on three design principles: blind evaluation (judges do not know which model produced which response), independent judgment (no inter-model discussion or debate), and consensus aggregation (rankings are pooled across models and repeated runs). This combination is deliberately chosen to avoid two well-documented failure modes in LLM-based evaluation: positional and verbosity bias from single-judge setups, and interaction contamination from multi-agent debate frameworks like ChatEval.
The RII is defined as a preference frequency metric, specifically how often a given model's responses are ranked first by peer judges. The paper notes that RII is invariant to linear scaling of ranking scores, which means it captures ordinal preference rather than absolute magnitude. That is an appropriate design choice given the subjectivity involved.
The study covers five models across five domains: programming, general knowledge, safety, logical reasoning, and mathematics. Each model acts as both generator and judge, creating a fully self-contained pipeline. With 25 prompts, 5 generators, 5 judges, and 5 repeated runs, the total evaluation count reaches 3,125, which provides reasonable statistical coverage for a study of this kind.
Methodology
The pipeline is well-structured. Responses are anonymised and randomised before presentation to judges, which is essential for suppressing brand recognition effects. Judges receive the original prompt alongside all anonymised candidate responses and must return a strict ordering in JSON format, with no ties permitted. Temperature is fixed at 0.3 across all models to balance reproducibility with response diversity.
One methodological detail worth noting is the self-judging problem. Each model evaluates its own response alongside competitors, but because responses are anonymised, a model has no direct way to identify its own output. Whether models can infer their own stylistic fingerprints from anonymised text is an open question the paper does not fully address, though it acknowledges the possibility of stylistic bias influencing judgments.
The rubric given to judges emphasises relevance, correctness where applicable, and clarity with conciseness. This is reasonable but somewhat underspecified. Different domains weight these criteria differently: correctness dominates in mathematics, while clarity and structure matter more in general knowledge responses. A domain-adaptive rubric might have produced more discriminating results, though it would also complicate cross-domain comparison.
Results and What They Show
The paper reports consistent preference patterns across domains, with certain models ranked highly by peers more frequently than others. The key findings, as summarised:
- Some models exhibit high preference scores with moderate consistency, suggesting their outputs are broadly appealing to peer judges.
- Others show low variance in rankings but also lower preference scores, indicating predictable but unremarkable output quality.
- Preference and consistency appear to be distinct dimensions of model behaviour, neither fully capturing the other.
- Patterns are stable across repeated runs, suggesting the RII signal is not dominated by stochastic noise at temperature 0.3.
The paper is appropriately cautious about what these results mean. Inter-model preference alignment is not the same as human preference, and it is certainly not a measure of objective correctness. The authors explicitly frame RII as a model-relative metric, which is the right framing. A model that consistently produces outputs preferred by other LLMs may simply be producing outputs that align with common training distribution biases, not outputs that real users would find most useful.
Limitations and Open Questions
The most significant limitation is the one the authors acknowledge most directly: the relationship between inter-model consensus and human judgment is assumed rather than demonstrated. Prior work cited in the paper suggests partial correlation between aggregated model preferences and human ratings, but partial correlation is a weak foundation for a benchmarking framework intended to serve as a practical evaluation tool. Without a calibration experiment comparing RII rankings to human preference data on the same prompts, the metric's external validity remains uncertain.
A second concern is the small prompt set. Twenty-five prompts across five domains means five prompts per domain. That is too few to draw strong domain-level conclusions, and the paper does not report confidence intervals or significance tests on domain-level RII differences. The aggregate signal across all domains may be stable, but domain-specific claims should be treated with caution.
The model selection also raises questions. The paper lists specific model versions including GPT-5.2 and Gemini Pro 3.1, which do not correspond to publicly known model releases at the time of writing. This creates a reproducibility concern: readers cannot verify which exact models were used, and the results cannot be independently replicated without access to those specific API versions.
There is also a structural circularity worth flagging. The framework measures what models prefer, then uses that preference as a proxy for quality. If the models share training data, RLHF preferences, or stylistic tendencies from common fine-tuning pipelines, the consensus signal may reflect shared biases rather than genuine quality differences. The paper acknowledges this, but does not propose a concrete method for disentangling preference from bias.
Despite these limitations, the framework addresses a real gap. As LLM outputs become harder to distinguish on correctness alone, comparative preference evaluation offers a practical complement to accuracy-based benchmarks. The blind peer ranking design is methodologically sound, the pipeline is reproducible in principle, and the RII metric is interpretable. Future work integrating correctness filtering and human calibration data could substantially strengthen the framework's validity claims.
The full paper is available at https://arxiv.org/abs/2607.21632.