Speculative decoding has become one of the more practically useful ideas in LLM inference research. The core insight is simple: use a cheap drafter model to propose token sequences, then verify them in parallel with the expensive target model. Tree-based extensions of this idea improve things further by organising proposals into branching candidate trees, so a single rejection does not invalidate everything downstream. The bottleneck that remains is the drafter itself: a weak drafter is fast but produces low-quality trees, while a stronger drafter improves quality but costs enough per step to erode the speedup you were trying to achieve in the first place.
TreeGraft (arXiv:2608.26112) addresses this directly by asking whether the same drafter must be used at every drafting step. The answer, predictably, is no, and the paper builds a coherent framework around that observation.
Key Contributions
TreeGraft introduces a two-drafter setup: a lightweight small drafter handles most steps, while a stronger middle drafter is called selectively. Both contribute to a single shared draft tree, which the authors call grafting. Three design questions organise the paper:
- Where to graft: Standard single-drafter tree construction restricts the candidate pool to leaf nodes from the previous step. TreeGraft expands this pool at middle-drafter steps, allowing the stronger model to rescore earlier nodes that the weak drafter may have under-ranked and reopen paths that were prematurely abandoned.
- How to graft: Existing hierarchical multi-drafter methods (such as Cascade Speculative Drafting and TriForce) are sequence-based and overwrite existing children when a stronger drafter generates new ones. In a tree setting, this is particularly destructive because overwriting a node discards entire subtrees. TreeGraft instead uses non-destructive grafting: middle-drafter nodes are inserted without removing existing branches, preserving paths the target model might still accept.
- When to graft: A lightweight online scheduler, distilled from an offline value system, decides at each drafting step whether calling the middle drafter is worth its cost. The value system predicts accepted length and log-latency for complete scheduling trajectories, and the scheduler is trained to regress the action-value margin between calling and skipping. At runtime, the scheduler uses only 17-dimensional tree signals and costs roughly 0.318 ms per call, well below the n-gram lookup cost and orders of magnitude below any neural forward pass.
Methodology
The small drafter in the main experiments is a training-free n-gram model that retrieves candidate tokens via longest-match lookup over the context, using target-model probabilities cached during prefill. This choice is deliberate: even the smallest neural models in the LLaMA 3 and Qwen3 families cost 29 to 45 ms per call, which would make the low-cost endpoint expensive enough to undermine the whole exercise. The n-gram drafter costs 0.92 ms per call.
The offline stage fits four independent MLP heads per candidate signal subset, predicting accepted length and log-latency under call and skip actions, conditioned on the current tree state and the future scheduling suffix. With a maximum of five drafting steps, exhaustive enumeration over all binary suffixes is tractable, so the margin target is exact rather than approximated. The online scheduler is then a three-layer MLP with 3,265 parameters that regresses this margin from the current state alone, without access to future suffixes.
One methodological detail worth noting: the paper explicitly separates seen and unseen model pairs during offline fitting and online evaluation, and holds MT-Bench entirely out of the offline stage. This gives a cleaner picture of generalisation than evaluations that reuse the same data throughout.
Results
Across 10 model pairs (LLaMA 3 and Qwen3 families) and 6 benchmarks, TreeGraft achieves a mean speedup of 1.60x over autoregressive decoding, compared with 1.32x for the all-small endpoint and 1.39x for the all-middle endpoint. The 15.1% average gain over the better fixed single-drafter strategy is the headline number. The maximum individual gain is 26.6% on Qwen3-32B with a 0.6B middle drafter on Alpaca.
The scheduler's adaptive behaviour across three qualitatively different regimes is informative:
- When the middle drafter dominates (e.g. LLaMA 3.3-70B with a 1B middle drafter, All Mid 2.30x vs All Small 1.44x), TreeGraft matches All Mid at 2.30x by calling the middle drafter at most steps.
- When the small drafter dominates (e.g. Qwen3-8B/0.6B, All Small 1.24x vs All Mid 0.85x), TreeGraft reaches 1.27x by calling the middle drafter sparsely, avoiding the severe slowdown of All Mid.
- When both endpoints are mediocre (Qwen3-32B/0.6B, All Small 1.21x vs All Mid 1.19x), TreeGraft achieves 1.47x by concentrating middle-drafter calls at selected steps rather than applying them uniformly.
The one case where TreeGraft falls behind is LLaMA 3.3-70B with an 8B middle drafter, where it trails All Mid by 1.6%. Given that All Mid is already the dominant strategy in that setting, this is a small and expected cost of adaptive scheduling.
The offline evaluation also includes a useful negative result: using the fitted value system directly as an online planner drops speedup to 0.95x despite a smaller offline gap. Small errors in predicted accepted length and latency compound when the planner optimises over future suffixes, which motivates the margin-distillation approach rather than direct deployment of the value system.
Limitations and Implications
Several limitations are worth flagging. The framework is evaluated only with an n-gram small drafter; replacing it with a small neural LLM would require a separate cost-quality study, and the authors acknowledge this. The value system and scheduler are fitted on LLaMA 3 and Qwen3, so behaviour on architecturally different models is unknown. The tree budget and expansion hyperparameters are fixed across all experiments, which may not be optimal for every setting.
More broadly, the paper is a good example of how inference-time efficiency gains can come from scheduling and composition rather than from training better components. The non-destructive grafting rule and the expanded candidate pool are both simple ideas that turn out to matter; the value-guided scheduler is the more novel piece, and the distillation approach to avoid deploying a future-conditioned model online is a clean solution to a real deployment problem.
For practitioners, the most immediately useful takeaway is that mixing a trivially cheap drafter with a moderately sized neural model, scheduled adaptively, can outperform either alone without any additional training. The code is available at the anonymous repository linked in the paper, and the full details are at arXiv:2608.26112.