The economics of LLM inference have pushed practitioners toward cascade architectures: route easy queries to a cheap small model, escalate hard ones to a larger, more capable model. The idea is sound, but the execution is typically fragile. Most deployed routers rely on raw confidence scores that are poorly calibrated, meaning the model's expressed certainty bears little relationship to its actual error rate. The consequence is that threshold selection becomes a manual, workload-specific exercise with no principled cost guarantee. UCCI (Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing) addresses this directly by making calibration the first-class design goal, then deriving the escalation threshold analytically from a cost objective.
Key Contributions
The paper makes three distinct contributions that are worth separating out:
- A calibration pipeline for token-level signals. UCCI extracts the margin between the top two token probabilities as a proxy for model uncertainty, then maps this to a per-query error probability using isotonic regression. This is a monotone, non-parametric fit that respects the ordinal structure of the confidence signal without assuming a parametric form.
- Theoretical grounding for threshold optimality. Under three stated assumptions, the paper proves that a threshold policy on the calibrated score is cost-optimal. The assumptions are not hidden; they are made explicit, which is more than most routing papers manage.
- Sample complexity bound for calibration error. Isotonic regression achieves O(n^{-1/3}) convergence in expected calibration error (ECE). This is a known result in the calibration literature, but applying it here gives practitioners a concrete handle on how much calibration data they need before the threshold guarantee holds.
Methodology
The routing signal is the token-level margin: the difference in log-probability between the most likely and second-most likely next token. High margin means the small model is confident; low margin triggers escalation to the large model. Raw margins are not reliable probability estimates, so UCCI fits an isotonic regression on a held-out calibration set to convert margins into calibrated error probabilities. Isotonic regression is a natural choice here because it enforces monotonicity without further assumptions, and it is computationally cheap compared to temperature scaling or Platt scaling variants.
The escalation threshold is then selected by solving a constrained cost minimisation problem: find the threshold that minimises expected compute cost subject to a quality constraint on the error rate. This is a clean framing. The cost model uses measured H100 latency rather than nominal API prices, which matters because latency profiles for 4B and 12B parameter models on the same hardware are not simply proportional to parameter count.
The evaluation workload is a production named entity recognition task with 75,000 queries, served by 4B and 12B instruction-tuned LLMs. The choice of NER is reasonable: it is a structured prediction task where token-level confidence is a natural signal, and F1 provides a calibrated quality metric. Whether the approach generalises to open-ended generation tasks where quality is harder to operationalise is a question the paper does not fully resolve.
Results
The headline numbers are competitive. UCCI achieves a 31% reduction in inference cost (95% CI: [27%, 35%]) at micro-F1 = 0.91, and reduces ECE from 0.12 to 0.03. The confidence interval is a welcome addition; too many systems papers report point estimates without acknowledging variance across query batches.
Against baselines, UCCI outperforms:
- Entropy thresholding, the most common uncalibrated baseline
- Split-conformal routing, which provides coverage guarantees but does not directly optimise cost
- A FrugalGPT-style learned threshold, which requires per-workload training and lacks the calibration step
The comparison against split-conformal routing is the most technically interesting. Conformal prediction methods have attracted attention for routing because they offer distribution-free coverage guarantees, but they do not natively optimise the cost-quality tradeoff. UCCI's calibration approach is less conservative and more directly aligned with the cost objective, which explains the performance gap.
Limitations and Open Questions
Several limitations deserve attention. The three assumptions underpinning threshold optimality are stated but not stress-tested empirically. If the calibration set is drawn from a different distribution than the production workload, the isotonic fit degrades and the threshold guarantee no longer holds. This is a standard covariate shift concern, but it is particularly acute for LLM workloads where query distributions can shift rapidly.
The evaluation is restricted to NER, a task with a well-defined token-level confidence signal and a scalar quality metric. Extending UCCI to tasks like summarisation, code generation, or multi-turn dialogue would require rethinking both the uncertainty signal and the quality constraint. Token-level margin is a reasonable proxy for extractive or classification-style tasks, but it is a weaker signal for tasks where correctness depends on long-range coherence.
The paper also does not address the cold-start problem: how many calibration queries are needed before the isotonic fit is reliable enough to trust the threshold? The O(n^{-1/3}) bound gives an asymptotic answer, but practitioners need finite-sample guidance. A sensitivity analysis over calibration set size would strengthen the practical case.
Finally, the two-model cascade is a simplification. Production systems often involve more than two tiers, and the optimal threshold structure for a three-model cascade is not a straightforward extension of the two-model case. The framework is theoretically extensible, but the empirical validation does not cover this setting.
These are not fatal objections. The core insight, that calibration should precede threshold selection rather than being treated as an afterthought, is correct and underappreciated in the routing literature. UCCI provides a clean, theoretically motivated instantiation of this principle with honest empirical evaluation on real hardware. The field has too many routing papers that simulate costs from global accuracy estimates; the decision to use actual model outputs and measured latency is the right methodological choice.
The full paper is available at arXiv:2605.18796.