Most LLM benchmarks in finance test whether a model can discuss financial concepts or generate plausible-looking code. Very few test whether an agent can actually run a backtesting engine, interpret its numerical output correctly, and report the right answer. That gap matters in practice: a misquoted Sharpe ratio or an understated maximum drawdown can lead portfolio managers to allocate capital to strategies that would fail under realistic conditions. Backtrader-Bench, accepted to the FinLLM Workshop at IJCAI 2026, is a direct attempt to close that gap.
What Problem Does This Address?
Existing financial LLM benchmarks such as PIXIU, FinBen, and InvestorBench cover NLP tasks and high-level decision-making. None of them measure strategy comprehension in the sense of correctly driving a backtesting library. The closest concurrent work, QuantCode-Bench, tests whether an LLM can produce executable trading code from a natural-language description, but stops short of asking whether the agent understands what that code actually produces when run.
Building a reliable benchmark for this capability faces three distinct obstacles. Static benchmarks risk data contamination once published. Numerical backtest outputs require ground truth derived from actual code execution, not human annotation. And automated question generation at scale tends to produce either trivially easy items or malformed ones that require expensive manual curation.
Key Contributions and Methodology
The framework has two complementary pipelines. The first is a deterministic MCQ pipeline. Given a configuration specifying a seed, ticker, strategy, and date range, the pipeline runs a backtest using the Python backtrader library, synthesises four-option multiple-choice questions from 33 templates across five trading strategies and three difficulty tiers (easy, medium, hard), and re-derives every answer with an independent checker that re-executes the same verification code. Because questions are generated at runtime under a configurable seed, no static answer key needs to appear anywhere public, which sidesteps the contamination problem directly.
The second pipeline is a generator-solver filtering system designed to mine harder questions automatically. A generator agent writes a free-form question together with verification code. If the code executes correctly and produces a verifiable answer, the question is converted to an MCQ with randomised distractors. A no-tools solver then attempts to answer it without code execution; questions it solves are discarded as too easy. A tool-augmented solver validates the remainder. The filter model used in experiments was GPT-5.4.
Evaluation covers 11 models without tools (10 independent runs each to account for stochasticity) and four with-tools configurations run through the Cursor Agent CLI, each operating in an isolated sandbox with identical permissions and timeouts.
Results
On the curated 30-question benchmark, the headline finding is clear:
- Tool-augmented agents (GPT-5.5 and Opus 4.7) reach 90.0% accuracy in a single pass.
- The best no-tools baseline (Opus 4.7 and Gemini 3.1 Pro, tied) averages 73.0% across 10 runs, a 17 percentage point gap.
- Composer-2 ranks last in both modes, suggesting weaker models gain less from tool access.
- Model rankings are not preserved across modes: GPT-5.3 Codex ranks third without tools but was not tested with tools, and the ordering among mid-tier models shifts substantially.
The mined question set is more revealing. Of 100 mining attempts, 98 produce well-posed questions; 38 pass the full pipeline (the no-tools solver fails, the tool solver succeeds). Accuracy drops substantially across all no-tools models on this harder set:
- Grok 4.3 falls from 59.3% to 21.1% (a 38-point drop).
- Kimi K2.5 drops from 55.7% to 15.8%.
- Three models score near the 25% random-chance baseline, statistically indistinguishable from guessing at n=38.
- Even the strongest no-tools models plateau around 60%, well below tool-augmented performance.
The error analysis points to specific failure modes. Agents most consistently err on multi-step derived metrics: maximum drawdown (which requires tracking a running peak across an equity curve), exposure-adjusted returns, and risk-adjusted ratios like Sharpe and Sortino. Bracket-order and trailing-stop questions produce the largest accuracy drops on the mined set, because they depend on backtrader's stateful event-loop and order-fill mechanics rather than any surface-level concept recognition.
Limitations and Open Questions
The paper is candid about several methodological constraints worth examining carefully.
First, with-tools evaluation is limited to a single pass per model configuration because each 30-question set takes 83 to 103 minutes to complete. This is a real problem: single-run variance on the mined set is substantial, with Opus 4.7 scoring 68.4% in one run and 52.6% in another on the same questions. Conclusions drawn from single-pass with-tools results should be treated as indicative rather than definitive.
Second, and more fundamentally, mined-question difficulty is relative to the filter model, not absolute. The filter was GPT-5.4, yet Gemini 3.1 Pro and GPT-5.5 still score 60.5% on questions that GPT-5.4 supposedly cannot solve without tools. This means the difficulty ceiling of the mined set is bounded by the filter's capability. As frontier models improve, the pipeline will need a stronger or composite filter to maintain meaningful separation between tool-requiring and tool-free questions.
Third, coverage is limited to backtrader. The framework's design is modular, but the templates and verification code are library-specific. Generalising to zipline, vectorbt, or other engines would require substantial additional template engineering.
Fourth, the paper evaluates models accessed through the Cursor SDK and CLI, which introduces a layer of abstraction between the raw model and the evaluation. It is not entirely clear how much the scaffolding, rather than the model itself, accounts for with-tools performance differences.
The longer-term ambition, using this MCQ infrastructure as a training corpus for reinforcement learning toward a domain-specific trading agent, is plausible in principle. The structured feedback each MCQ provides (correct answer, verification code, difficulty tier) is exactly the kind of signal RL fine-tuning benefits from. Whether the resulting agent would generalise beyond backtrader or beyond the five strategies covered here remains an open question that the paper does not yet address.
For researchers working on financial AI evaluation or tool-augmented agents more broadly, Backtrader-Bench represents a methodologically careful step toward benchmarks that test genuine computational reasoning rather than surface fluency. The contamination-resistance mechanism is elegant, the filtering pipeline is a practical solution to the quality-at-scale problem, and the error analysis offers actionable signal about where current models actually fail. The full paper is available at arxiv.org/abs/2608.11232.