The operational cost of running LLM-based agents at scale is, at this point, a genuine engineering problem rather than a theoretical concern. Inference spend compounds quickly when agents loop, re-send stable context without caching, or route simple tasks through heavyweight reasoning models. What has been largely absent from the tooling ecosystem is a principled, offline-capable profiler that treats token expenditure with the same rigour that CPU profilers apply to compute cycles. Wattage, released on GitHub by Faizan Raza, is a credible attempt to fill that gap.
Architecture and Data Model
Wattage ingests OpenTelemetry JSON trace exports conforming to the GenAI semantic conventions and normalises them into a five-level hierarchy: sessions, tasks, loops, iterations, and calls. This is a sensible decomposition. Agent traces are not flat sequences of LLM calls; they have recursive loop structure, and a profiler that flattens them loses the information needed to distinguish a healthy retry from a pathological oscillation. By preserving loop nesting, Wattage can reason about progress across iterations rather than just pricing individual calls in isolation.
The tool ships with a vendored, dated pricing snapshot (the sample output is stamped 2026-07-18-verified), which means cost figures are reproducible and auditable. This is a meaningful design choice. A tool that queries a live pricing API introduces non-determinism into CI pipelines; a committed snapshot does not. The tradeoff is that the snapshot requires maintenance as model pricing changes, but that is a tractable operational burden compared to the alternative of non-reproducible CI failures.
Critically, Wattage refuses to fabricate numbers. An unpriced model leaves that call's cost at zero and triggers a loud CI failure with exit code 4. This is the correct engineering stance. Silent underestimates in cost accounting are considerably more dangerous than loud failures, because they produce false confidence in budget projections.
The Eight Detectors and Their Coverage
The detector suite covers eight distinct waste patterns:
- prefix_churn: stable context re-sent rather than cached, the most common and most directly addressable source of avoidable spend.
- cache_gap: caching is attempted but subsequent reads fail to redeem it, suggesting cache key mismatches or prompt mutations between calls.
- verbosity: output token counts disproportionate to the complexity of the step, which can indicate poorly constrained output schemas or missing
max_tokensbounds. - redundant_tool_calls: repeated invocations of the same tool, detected via both exact and fuzzy matching.
- nonconvergence: the most technically interesting detector, discussed separately below.
- retrieval_thrash: repeated retrieval that returns no relevant results, a signal that the retrieval strategy or query formulation is broken.
- model_mismatch: expensive model capacity applied to work a cheaper model could handle.
- reasoning_overspend: heavy reasoning-token spend on steps that do not warrant it.
The quality_risk tagging system is particularly well-considered. Detectors that recommend changes with plausible quality implications, such as model downgrades or reduced reasoning budgets, only contribute to the efficiency score when a --quality map backs them with empirical evidence. This prevents the tool from optimising for cost at the expense of task performance, which would be the failure mode most likely to erode trust in automated cost gates.
The Convergence Engine: Methodology and Benchmarking
The nonconvergence detector is where Wattage makes its most substantive technical claim. The problem it addresses is well-known in the agent research community: naive duplicate detection based on exact string matching or hash comparison misses a wide class of non-convergent behaviours. An agent retrying with a fresh timestamp each iteration, oscillating between two strategies, or producing unique outputs that carry no new information all evade SHA-256 exact-match detection. These patterns are not edge cases; they are characteristic failure modes of agents operating under ambiguous stopping conditions or broken tool feedback loops.
Wattage benchmarks its convergence classifier against a SHA-256 exact-match baseline on a hand-reviewed set of ten labelled synthetic loops. The reported results are striking:
- Wattage classifier: Precision 1.00, Recall 1.00, F1 1.00
- SHA-256 exact-match: Precision 1.00, Recall 0.14, F1 0.25
The recall gap is the key finding. Exact-match catches only 14% of the non-convergent loops in the benchmark set, which is consistent with what one would expect given that most real-world thrashing involves semantically similar but syntactically distinct calls. Wattage's classifier, presumably operating on semantic similarity or structural features of the call sequence, catches all of them.
The honest caveat here is that ten labelled examples is a small benchmark. The synthetic nature of the loops means the classifier may be tuned to patterns that were anticipated during construction rather than patterns that emerge organically in production traces. The authors do provide a reproducible benchmark harness (uv run python -m benchmarks.harness), which is the right move, but independent validation on a larger and more diverse trace corpus would substantially strengthen the claim. This is a reasonable limitation for an early-stage tool, not a fundamental objection.
The prefix_churn fix simulation on a genuine three-turn demo trace shows a 44.7% cost reduction from enabling prompt caching on the stable prefix. The dollar figures are small by design, since it is a minimal trace, but the mechanism is sound and scales linearly with trace length.
CI Integration and the Baseline Update Problem
The CI integration design is thoughtful. The wattage ci command accepts threshold expressions like score_below:80,cost_delta_pct_above:5,any_critical:true, which gives teams fine-grained control over what constitutes a regression. The baseline is a committed JSON file rather than a database or external service, which keeps the gate stateless and portable across CI providers.
The documentation correctly identifies the baseline update problem as a non-trivial operational concern. A PR job runs on a throwaway checkout and cannot commit back to the repository, so a second workflow triggered on push to the default branch is required to update the baseline after each successful merge. Skipping this step means every PR compares against a stale baseline, which eventually renders the gate meaningless. This is a common failure mode in any CI system that maintains committed state, and it is good that the documentation addresses it explicitly rather than leaving it as an exercise for the operator.
The SARIF output is a practical addition. GitHub's Security tab is a natural place to surface cost regression findings, since it is already the destination for static analysis and dependency audit results. Treating token waste as a category of technical debt alongside security vulnerabilities is a framing that aligns with how engineering teams actually prioritise remediation work.
Positioning and Broader Context
Wattage sits at the intersection of two maturing areas: LLM observability and agent evaluation. Tools like LangSmith, Phoenix, and Weave address trace collection and qualitative evaluation, but cost regression as a first-class CI gate is less well-served. The closest analogue in traditional software engineering is performance regression testing, where tools like Bencher provide similar gate semantics for latency and throughput. Wattage applies that pattern to token spend, which is the dominant cost dimension for agent workloads.
The extensible detector architecture, using Python entry-point groups so that custom detectors can be added without forking the core pipeline, is the right design for a tool that needs to accommodate the diversity of agent architectures in production. Different agent frameworks produce different waste patterns, and a fixed detector set will inevitably miss domain-specific inefficiencies.
The offline-first, no-API-key design is worth emphasising. Many observability tools in this space require sending traces to an external service, which creates data residency concerns for teams working with sensitive prompts or proprietary agent logic. Wattage's ability to run entirely locally against a local trace file removes that barrier entirely.
The tool is early-stage, the benchmark corpus is small, and the pricing snapshot will require ongoing maintenance. But the core abstractions are sound, the methodology is transparent and reproducible, and the CI integration addresses a real gap in the agent development workflow. For teams operating LLM agents at any meaningful scale, a cost-regression gate of this kind is not a luxury; it is the same category of discipline that performance budgets and memory limits represent in systems engineering. Wattage provides a concrete starting point for that practice.