Speculative decoding has become one of the most practically significant techniques for reducing latency in large language model inference. By using a smaller, faster draft model to propose candidate tokens that a larger target model then verifies in parallel, it can achieve substantial throughput gains without altering the target model's output distribution. The efficiency gain depends entirely on the average accepted length, the number of draft tokens that survive verification per step. A new paper, Mistletoe: Stealthy Acceleration-Collapse Attacks on Speculative Decoding, identifies a previously overlooked attack surface in this mechanism and demonstrates that an adversary can silently destroy those efficiency gains while leaving visible output quality intact.
This matters because speculative decoding is now deployed widely, including in production inference stacks for models like Llama and Gemma. Security analyses of LLM systems have focused almost exclusively on output-level attacks, prompt injection, jailbreaks, or poisoning that changes what a model says. Mistletoe is a different kind of threat: it targets the mechanism by which inference is accelerated, not the content of the inference itself.
The Core Vulnerability
The attack surface arises from an inherent property of model-based speculative decoding: the draft model is trained to approximate the target model's token distribution, but this approximation is never perfect. The gap between the two distributions is what the authors call the drafter-target mismatch, and it is this gap that Mistletoe exploits.
Speculative decoding's verification step uses a rejection sampling scheme. If the draft model assigns high probability to the same tokens the target model would choose, most draft tokens are accepted and throughput improves. If the draft and target distributions diverge, tokens are rejected and the system degrades toward standard autoregressive decoding, with no speedup and added overhead. The key insight is that an attacker who can perturb the input (for example, through adversarially crafted prompts or documents in a retrieval-augmented generation pipeline) can widen this distributional gap without necessarily changing the target model's final output in a way that a user would notice.
Methodology
The Mistletoe attack jointly optimises two objectives. The first is a degradation objective, which increases the KL divergence or distributional distance between the draft model's predictions and the target model's predictions on the perturbed input. The second is a semantic-preservation objective, which constrains the target model's own output distribution to remain close to what it would produce on the clean input. The tension between these two objectives is the central technical challenge: gradients that push the draft model away from the target can also shift the target's outputs, which would make the attack detectable.
To resolve this, the authors introduce a null-space projection mechanism. Degradation gradients are projected into the null space of the semantic-preservation constraint, meaning the perturbation is steered to maximally disrupt draft-target agreement while remaining orthogonal to directions that would alter the target model's visible behaviour. This is a clean formulation, drawing on constrained optimisation ideas that appear in other adversarial ML contexts but applied here to a novel objective.
The attack is evaluated across several speculative decoding systems, including configurations with different draft model sizes and architectures. Perturbations are generated at the token embedding level, consistent with a threat model where an attacker controls some portion of the input context.
Results and What They Mean
The reported results are striking. Mistletoe substantially reduces average accepted length across tested systems, which directly collapses the speedup ratio. Token throughput drops significantly, in some configurations approaching or reaching the baseline autoregressive decoding rate, meaning the speculative decoding overhead is incurred with none of the benefit. Critically, perplexity and output quality metrics remain largely unchanged, consistent with the semantic-preservation objective working as intended.
Key findings from the paper:
- Average accepted length can be reduced to near-minimum values, effectively disabling the acceleration benefit of speculative decoding.
- The attack transfers across different draft model configurations, suggesting it is not narrowly tuned to a single system.
- Output perplexity and quality scores remain stable, making the attack difficult to detect through standard monitoring of model outputs.
- The null-space projection mechanism is empirically validated as essential; without it, the degradation objective bleeds into the target model's outputs and the stealthiness property breaks down.
Limitations and Open Questions
Several aspects of the threat model deserve scrutiny. The attack as described requires the ability to craft or inject adversarial text into the model's input context. This is a realistic scenario in RAG systems, shared inference endpoints, or multi-turn conversations where user-controlled content influences the context window, but it is not universal. The paper does not extensively address defences, which is a significant gap. Whether input sanitisation, draft model ensembling, or online monitoring of acceptance rates could mitigate the attack is left largely unexplored.
The transferability results are promising but partial. It would be useful to know how well perturbations crafted against one draft model architecture transfer to a different one, particularly in black-box settings where the attacker does not know which draft model is in use. The current evaluation appears to assume some knowledge of the target system's configuration.
There is also a broader question the paper raises but does not fully answer: how much of this vulnerability is specific to speculative decoding versus a general property of any system that relies on a learned approximation to a larger model? Medusa-style multi-head drafting, EAGLE, and other variants all depend on similar drafter-target alignment, and it would be worth understanding whether Mistletoe-style attacks apply uniformly or whether some architectures are more resilient.
The work is a useful reminder that efficiency techniques introduce their own attack surfaces. As speculative decoding becomes infrastructure rather than a research novelty, understanding its failure modes under adversarial conditions is not optional. Mistletoe is a well-motivated and technically careful contribution to that understanding, even if the defensive side of the story remains to be written.
The full paper is available at arXiv:2605.14005.