When you deploy an LLM-based agent, how do you know it is actually doing the right things, not just producing the right answers? Most production systems answer this with an outcome judge: feed the original request and the final reply to a model, ask if the case was handled well, and use that verdict to gate releases or shape reward signals. A new paper, trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories, puts that default practice under a controlled microscope and finds a structural blind spot that pooled evaluation metrics routinely obscure.
The core problem is intuitive once stated. An agent that issues a refund of the correct amount without ever checking eligibility looks, from the outside, identical to one that followed every required step. Outcome-only evaluation cannot distinguish them by design. The paper measures exactly how large this gap is, and tests whether alternative judge designs close it.
Methodology: Correct-by-Construction Ground Truth
The experimental design is the paper's most valuable contribution. Rather than relying on human annotation, which merely relocates the measurement problem, the authors construct a deterministic support-desk environment solved by a scripted oracle policy that is provably correct. A fault injector then breaks exactly one thing at one known step and replays the trajectory so internal observations remain consistent. Every trajectory therefore carries an exact label at zero annotation cost: faulty or not, at which step, of which type, and crucially, whether the customer-visible outcome survived the fault.
That last label is the key stratification. Faults are classified as loud (the wrong answer reaches the customer) or silent (the process broke but the answer survived). This distinction is what makes the measurement possible and is precisely what pooled recall averages away.
Five judge designs are compared across 400 trajectories (100 clean, 175 silent-fault, 125 loud-fault):
- A programmatic rule engine
- An outcome-only LLM judge (14B model)
- A step-rubric LLM judge at two model sizes (14B and 8B)
- A self-consistency ensemble (k=3, 14B)
Judges are scored on detection recall, step localisation, fault typing, calibration (ECE), and wall-clock cost per trajectory.
Key Findings
The headline numbers are striking and the stratification is what makes them legible.
- The outcome-only judge catches 84% of loud faults but only 45% of silent ones, while producing false alarms on 33% of clean trajectories. Pooled, this looks like 61% recall, a number that would pass a cursory review.
- The step-rubric judge (same 14B model, different view) reaches 77% silent recall with zero false alarms across 100 clean trajectories. Every one of its 257 flagged trajectories was genuinely faulty.
- Step localisation for the 14B step judge is near-exact (97.3% conditional on detection), and its calibration is substantially better (ECE 0.033 vs 0.253 for the outcome judge).
- The rule engine is free, perfectly localised, and never false-alarms, but misses more than half of silent faults (42.9%). Two of the six fault types are simply outside what deterministic rules can express.
- The 8B step judge flags 397 of 400 trajectories including all clean ones. It is, in the authors' words, "an always-say-faulty baseline wearing a judge's clothes." Its perfect silent and loud recall is meaningless against a 100% false-alarm rate.
- Self-consistency ensembling (k=3) triples cost to 30 seconds per trajectory and improves nothing on any quality axis relative to a single greedy pass of the 14B step judge.
One fault type, unsupported_claim, deserves particular attention. An invented promise appended to an otherwise perfect trajectory evades the rule engine entirely (by design, since rules cannot read semantic content), evades the step judge 82% of the time, and is the one case where a frontier model might plausibly do better than the local models tested here. The authors are transparent about this limitation.
Detection Versus Attribution
A finding that will matter for anyone routing judge verdicts to dashboards or ticket queues: detecting a fault and correctly naming it are substantially different problems. The 14B step judge achieves F1 of 0.923 on detection but only 0.606 macro-F1 on fault typing. Concretely, it finds all 50 hallucinated_argument cases but calls 35 of them wrong_tool, a defensible misreading given that fetching a policy for an invented SKU is also a tool that does not serve the sub-goal. premature_stop scatters worse across three categories.
The practical implication is that headline detection numbers should not be assumed to transfer to attribution tasks. If judge verdicts are used to categorise failures for process improvement, attribution accuracy needs its own evaluation.
Implications and Limitations
The paper's argument is methodological as much as empirical. The authors contend that any judge evaluation that does not stratify recall by outcome survival is hiding the faults that process supervision exists to catch. This is a reasonable position and the data support it clearly.
The step-level approach comes at a cost: roughly 3x the latency of outcome-only judging (10.4 vs 3.5 seconds per trajectory in the local model setup). Whether that cost is acceptable depends on the deployment context, but the authors make a fair case that for release gating or reward shaping, the outcome-only judge's false-alarm rate alone (33% of clean trajectories flagged) is disqualifying regardless of the silent recall problem.
The limitations are stated honestly. This is one domain, a support desk with encoded preconditions, which is a friendly case for step-level judging. Open-ended coding or browsing agents have no comparable rule engine, and the authors acknowledge that domains without them are likely worse, not better. Each injected fault breaks exactly one thing at one step; real failure cascades are messier. All experiments use local models (Qwen2.5:14B and Llama3.1:8B), so frontier judge performance on the unsupported_claim gap is an open question. The judged set is not stratum-balanced, and "silent" is defined by what the environment can observe, which may not match what an auditor cares about.
The reproducibility commitment is commendable. Raw verdicts are released alongside an analysis pipeline that regenerates every table and figure without a model call. For a paper making methodological claims about evaluation practice, that matters.
The broader takeaway sits in a tradition of process-versus-outcome supervision work from mathematics reasoning (Lightman et al., 2024; Uesato et al., 2022) now applied to the evaluator side of agent deployment. The finding that the blind spot is structural rather than a model capability failure, since the same 14B model closes most of the gap when shown the full trajectory, should inform how teams design monitoring infrastructure for tool-using agents in production.
Read the full paper at arxiv.org/abs/2609.00038.