Evaluating language model agents at scale has become one of the more quietly painful problems in AI research. The benchmark count has exploded, with over 200 agentic suites appearing since 2024 alone, but each one ships its own environment, action space, and scoring logic. Running m benchmarks against n agents naively requires O(mn) integrations, a combinatorial burden that in practice forces researchers to cherry-pick a handful of popular benchmarks and hope the results generalise. This paper, Harbor Adapters and Harbor-Index, attacks that problem directly with a standardised integration layer and a curated meta-dataset designed to make large-scale agentic evaluation tractable.
Key Contributions
The paper makes three distinct contributions that build on each other logically. First, Harbor Adapters extends the Harbor Python library (originally built for Terminal-Bench) into a general-purpose evaluation infrastructure. Each benchmark gets a single adapter that exposes its tasks, environments, and scoring through a shared schema of instruction, environment, tests, and solution. Each agent gets a single Harbor integration. The result is O(m+n) complexity rather than O(mn), and as of May 2026 the system supports 22 agents and over 80 benchmarks, including both natively agentic suites like SWE-bench and non-agentic ones that the adapters make agentic by exposing tool use and executable environments.
Second, the authors conduct what is probably the most systematic cross-benchmark agent evaluation published to date: 8 models across capability tiers from Google, OpenAI, and Anthropic, each run under two harnesses (the cross-family Terminus-2 and one vendor-native harness), across 54 benchmarks and 6,627 tasks, with three trials per configuration. Total compute exceeded $300K and 226B tokens. The scale is only feasible because of the adapter infrastructure.
Third, they introduce Harbor-Index 1.0, a curated set of 82 tasks drawn from 29 benchmarks, filtered for difficulty, diversity, and quality through a combination of automated difficulty scoring, AI auditing, and human review with an audit-and-fix loop. No evaluated model-harness configuration exceeds 30% pass rate, with the strongest (GPT-5.5 with Codex) reaching 28.0%.
Methodology and Validation
The adapter validation methodology deserves attention. Parity experiments run the Harbor-adapted benchmark against the original implementation under matched conditions (same agent, model, tools, prompt template, decoding parameters) for k=3 trials, reporting mean and standard error. Adapters that fail parity are debugged and re-run; irreducible differences due to non-deterministic external services are documented per benchmark. This is more rigorous than most benchmark reimplementations in the literature, where parity is often assumed rather than verified.
The large-scale evaluation uses a linear mixed model with benchmark as a random intercept to control for the dominant effect of task difficulty (ICC = 0.75) when comparing model and harness effects. This is methodologically appropriate and avoids the naive mistake of comparing raw scores across benchmarks with very different difficulty distributions.
Harbor-Index construction applies several quality filters: difficulty filtering to exclude tasks already near saturation, AI audit to flag ambiguous or broken tasks, human audit for a subset, and an audit-and-fix loop to repair salvageable tasks rather than simply discard them. The resulting index is designed to remain challenging as models improve, with the authors committing to maintain it as a live benchmark.
Main Findings
The empirical results yield several findings worth unpacking:
- Model capability dominates harness design. The fixed-effect range across models (0.451) is 5.2 times the range across harnesses (0.087) in the mixed model. Harness choice matters at the margins but does not substitute for base-model capability. Six of eight model coefficients are statistically significant versus two of four harness coefficients.
- The benchmark space is low-dimensional. PCA of the 16x54 score matrix shows most variance is explained by a single shared capability factor, with agent scaffolding adding one additional direction. Greedy selection on Spearman rank correlations finds that after 12 benchmarks are selected, every remaining benchmark correlates at rho >= 0.7 with a selected one. The practical implication is that many benchmarks are measuring overlapping capabilities.
- Difficulty and uniqueness are orthogonal. The benchmarks that provide independent ranking signal span the full difficulty spectrum rather than clustering at the hard end. Some difficult benchmarks (BigCodeBench) are largely predictable from others; some easier ones testing specialised capabilities (FinanceAgent, LabBench) resist prediction. This is an important design principle for future benchmark construction.
- Frontier models are more token-efficient but not cheaper. Frontier models use fewer tokens at every difficulty tier, with the relative gap widest on easy tasks (42% of weaker-model token usage in the easiest bucket). However, higher per-token pricing means other models cost 2-3 times less per trial. The frontier value proposition is most pronounced on medium-to-hard tasks where absolute performance gains are largest.
- Task failures are a significant confound. Among unresolved tasks, failures attributable to misspecified or broken task definitions are common. This is a known problem in the benchmark literature but the scale of the analysis here makes it more visible: a substantial fraction of apparent model failures may reflect task-design flaws rather than genuine capability limits.
The failure mode analysis on 6,028 trajectories (with human annotation on 200 and LLM-as-judge extension to the full set, inter-rater kappa = 0.66 for humans) finds that wrong factual answers, algorithmic bugs, and hidden-test regressions dominate across all frontier models. Harness design shapes how models fail: native harnesses like Claude Code and Codex allow iterative self-correction, while Terminus-2's linear Plan-Execute-Complete structure is more vulnerable when refinement is needed. Gemini's heavy reliance on external search causes a distinct failure mode the authors call "information drift", where the model replaces an initially correct hypothesis with conflicting online information.
Limitations and Implications
The authors are candid about scope. The large-scale evaluation covers 54 of the collected benchmarks (cost and adapter availability limit the rest), a focused set of models from three major providers, and two harness families. The findings are not universal across all LLMs or agent architectures. The empirical difficulty metric is configuration-dependent and can be distorted by task brittleness and evaluation setup, so it is a proxy rather than a ground truth for intrinsic difficulty.
The concern about reward hacking is worth taking seriously. The paper notes that stronger future agents may exploit benchmark artifacts more effectively than current ones, which is a standard limitation of any fixed evaluation set. The commitment to maintaining Harbor-Index as a live benchmark with periodic updates is the right response, though it raises questions about longitudinal comparability that the authors do not fully address.
The broader implication for the field is that benchmark proliferation without infrastructure investment has real costs. The O(mn) integration problem is not just an engineering inconvenience; it actively distorts what gets evaluated and reported. Harbor Adapters is a concrete proposal for how to address that, and the open-source release of adapters, evaluation results, trajectories, and Harbor-Index makes it possible for others to build on rather than replicate this work.
For researchers designing new benchmarks, the redundancy analysis is worth reading carefully. The finding that 3 representative tasks per benchmark recover system rankings at mean rho = 0.923 suggests that many existing benchmarks contain far more tasks than are needed for reliable model differentiation, at least within the capability range currently being evaluated. Whether this holds as models improve is an open question, but it has immediate implications for evaluation cost.
Full details, open-source artifacts, and interactive results are available at arXiv:2609.04298 and harbor-index.org.