Prime Intellect recently published results from what is arguably the most systematic head-to-head evaluation of frontier AI coding agents to date. The nanoGPT speedrun benchmark asked 18 models, running autonomously for up to 9 days, to optimize a training run for a small GPT model, measured by validation loss on a fixed token budget. The task is well-scoped enough to be reproducible yet open-ended enough to demand genuine research-level reasoning about optimizer design, hyperparameter tuning, and architectural choices. With 153 total runs and curated traces including tool calls and scratchpads, this is a rare dataset for studying how capable these systems actually are at iterative empirical research.
The human record sits at a validation loss of 2,600 (lower is better), with a baseline of 3,290. The best any agent achieved was Fable 5 at 2,726, closing 81.7% of the gap between the baseline and the human record. That is genuinely impressive, but the distribution of results below it is equally informative. Opus 5 closed 53.6%, Kimi K3 (via prime-agent) closed 52.2%, and the rest of the leaderboard drops off fairly steeply. A cluster of models sits in the 8-12% range, which means they barely moved beyond what a competent but unsophisticated optimizer configuration would achieve.
What the Task Actually Measures
The nanoGPT speedrun is not a pure coding task. It requires an agent to form hypotheses about what changes to a training configuration or optimizer will reduce validation loss, implement those changes, run experiments, interpret results, and iterate. This is closer to the workflow of a machine learning engineer doing ablation studies than it is to writing a CRUD application. The evaluation therefore probes a qualitatively different capability than most coding benchmarks: the ability to reason about empirical feedback loops in a scientific context.
This matters for interpreting the results. A model that writes syntactically correct code but lacks intuitions about optimization dynamics will struggle here even if it scores well on HumanEval or SWE-bench. The task penalizes models that cannot prioritize which experiments to run first, since compute and time are finite. Fable 5 consumed 800M total tokens and ran for 8.7 days, while GPT-5.6 Sol consumed 2.9B tokens over 6.1 days and achieved a worse record. Token efficiency is not the only axis, but the data suggests that throwing more compute at the problem is not sufficient without correspondingly better reasoning about which directions are worth exploring.
The Efficiency Question and What the Equal-Budget Comparison Reveals
Prime Intellect's equal-budget comparison is methodologically important. By normalizing each model's best final run to a 24-hour budget and reporting the best validated record within that constraint, they disentangle raw capability from the willingness to keep running. Several models that look mediocre on the raw leaderboard perform comparably to higher-ranked models when budgets are equalized. GPT-5.6 Sol Pro, for instance, reaches 3,100 within 24 hours, which is competitive with Kimi K3's 3,125 despite Kimi K3 holding a higher overall rank after longer runs.
This has practical implications. If you are deploying an agent for a time-constrained research task, the 24-hour-budget ranking is more relevant than the unconstrained ranking. The two orderings do not agree perfectly, which means that some models are better at rapid iteration and early gains while others are better at sustained, longer-horizon exploration. Fable 5's dominance is robust across both views, but the middle of the distribution is more fluid than the headline leaderboard suggests.
Token consumption also varies by nearly two orders of magnitude across models. Grok 4.5 achieved a record of 3,120 using only 46M total tokens, while GPT-5.6 Sol used 2.9B tokens for a slightly worse record of 3,042. This is a striking difference. Either Grok 4.5 is making better use of each token, or it is exploring a narrower region of the search space. Without access to the full traces it is difficult to distinguish between these, though Prime Intellect does provide 41 curated trajectories for inspection.
Autonomous Research Agents and the Serial vs. Parallel Distinction
Several runs are tagged with a "serial era" label, indicating that the agent executed experiments sequentially rather than in parallel. This is a meaningful architectural distinction. A serial agent must commit to a direction before seeing the results of alternative branches, which limits its ability to hedge against uncertainty in the search space. A parallel agent can run multiple hypotheses simultaneously and prune based on early results, which is closer to how a human research team operates.
The serial era runs include Opus 5, Kimi K3 (prime-agent), GPT-5.6 Sol Pro, GPT-5.6 Terra, and GPT-5.5. Their performance is not uniformly worse than non-serial runs, which suggests that the quality of the search strategy within a serial framework can compensate for the lack of parallelism, at least on this task. Opus 5's serial run at 53.6% gap closed is second only to Fable 5. That said, the task's structure, where each experiment takes fixed compute, likely means that serial agents are operating under a harder constraint than the labels alone convey.
The broader point is that this benchmark begins to expose the architectural choices that matter for autonomous research agents: not just which base model is used, but how the agent harness structures exploration, manages context across long horizons, and decides when to exploit a promising direction versus explore new ones. These are questions that single-turn benchmarks cannot answer.
Limitations and What Remains Unknown
Several limitations deserve attention before drawing strong conclusions. First, the benchmark is a single task domain. Optimizer tuning for a small language model is a specific niche within ML research, and performance here may not generalize to other empirical research tasks, let alone to software engineering, mathematical reasoning, or scientific discovery more broadly. The correlation between nanoGPT speedrun performance and general research capability is an open empirical question.
Second, the harness configuration varies across models. Fable 5 runs with claude-code at a "high" compute tier, while many others run at "max" or "xhigh". The number of experiments, API calls, and days also differs substantially. These are not controlled variables, which makes it difficult to attribute performance differences solely to the underlying model. A fairer comparison would hold harness configuration constant across models, though this may be impractical given the proprietary nature of some systems.
Third, several runs are still marked as running at the time of publication, including Qwen3.8 Max, DeepSeek V4 Pro, Grok 4.6, Muse Spark 1.2, and GLM 5.3. GLM 5.3 has no record at all yet. The leaderboard is therefore a snapshot rather than a final result, and rankings could shift. This is worth keeping in mind when comparing models near the bottom of the table.
Finally, the human record itself is a moving target. The benchmark measures gap closed relative to a specific human achievement, but that achievement was itself the result of a competitive speedrun community iterating over time. The framing of "81.7% of the human record gap closed" is compelling, but it obscures the fact that the human record was set under different constraints, with different tools, and with community knowledge accumulated over many runs. A direct head-to-head comparison between a frontier agent and a skilled human working the same task from scratch, with the same time budget, would be more informative.
What This Signals for the Near Term
The gap between the best agent and the human record is still meaningful. Closing 81.7% of the gap is not the same as matching human performance, and the remaining 18.3% likely reflects the hardest, most insight-dependent improvements. The distribution of results also suggests that most frontier models are nowhere near saturating this benchmark. A median model closes roughly 20-30% of the gap, which leaves substantial room for improvement.
What is most significant about this work is the methodology rather than any individual result. Running 153 autonomous experiments across 18 models, with full traces, on a well-defined empirical research task, is the kind of systematic evaluation that the field has lacked. As AI agents are increasingly deployed for scientific and engineering research tasks, benchmarks that measure iterative, hypothesis-driven problem solving will matter more than those that measure single-shot code generation. Prime Intellect's speedrun framework is a credible step in that direction, and the public release of traces provides a resource for studying agent behavior at a level of detail that most evaluations do not.
The question worth watching is whether the gap between Fable 5 and the human record closes in the next generation of models, or whether it proves to be a ceiling that requires architectural changes beyond scaling the base model. Given how rapidly the top of this leaderboard has moved, the answer will probably come sooner than expected.