One of the more puzzling failure modes in large language model reasoning is the gap between apparent capability and actual performance. A model can demonstrate mastery of a concept in one context and then fail on a structurally similar problem elsewhere. The standard explanation is that these are hard problems at the edge of the model's competence. A new paper, Woodpecker Distillation: Weak Models Diagnose Reasoning Bugs in Strong Models, challenges that framing. The authors argue that many such failures are not evidence of global incompetence but are instead localized, recoverable errors in intermediate reasoning steps. That distinction turns out to matter quite a lot for how you might fix them.
The Core Observation
The paper begins with a clean empirical observation. Take a strong model that has failed on a reasoning problem under greedy decoding. Extract an intermediate prefix from its failed trajectory. Now insert a short patch generated by a weaker model at that prefix, and let the strong model continue from there. Surprisingly often, the strong model then arrives at the correct answer.
The authors test this A-B-A intervention scheme using Qwen3-4B-Instruct-2507 as the strong model and Gemma-3-4B-IT as the weak probe, evaluated across five mathematical reasoning benchmarks including AIME 2024, AIME 2025, MATH-500, Olympiad, and Omni-Hard. On problems where the strong model fails outright, patched continuations improve Pass@16 by roughly 2 percentage points overall, with gains of nearly 5 points on AIME 2025 and over 3 points on Omni-Hard.
This is the phenomenon they term a reasoning bug: a recoverable failure where the current trajectory is headed toward an incorrect answer, but a short local intervention can redirect subsequent reasoning toward a correct one. The definition is deliberately operational. It does not require pinpointing a single erroneous token; it only requires that the failure be locally repairable.
Why Naive Imitation Fails
The obvious follow-up is to just train the strong model on these repaired trajectories. Fine-tune on the weak model's patches, or on the full corrected traces, and the problem should go away. The paper shows this does not work reliably.
The authors test two imitation baselines: training the strong model to reproduce the weak model's patch text, and training it on the complete trajectory obtained after inserting the patch. Both underperform the intervention itself by a substantial margin. Their interpretation is that the useful signal is not in the surface text of the patch. The weak model's words are not the thing being transferred. What matters is how the patch reshapes the strong model's future token distribution after the same prefix. Copying the patch text teaches the model what the weak model said; it does not teach the model why that local nudge was effective or how to replicate the downstream effect in its own reasoning.
This is a genuinely important distinction. It connects to a broader issue in distillation research: token-level imitation of teacher outputs does not always transfer the teacher's functional knowledge, particularly when the teacher and student have different underlying representations or reasoning styles.
Woodpecker Distillation: Methodology
The proposed method addresses this by treating the weak model not as a teacher to imitate but as a diagnostic probe. The training pipeline works as follows:
- For a given strong-model prefix, sample multiple short patches from the weak probe model.
- For each patch, let the strong model continue and evaluate the outcome using a verifier.
- Divide patches into positive (outcome correct) and negative (outcome incorrect) sets.
- Compare the future token distributions induced by positive versus negative patches under a frozen copy of the strong model (the reference model).
- Construct a contrastive teacher distribution that upweights tokens associated with successful interventions.
- Train the trainable copy of the strong model to match this teacher distribution on the original, unpatched context.
The key move is that the strong model is trained on its own unpatched prefix, not on the weak model's text. The weak model's patches are used only to generate contrastive evidence about which future token distributions are associated with successful versus unsuccessful local repairs. The trainable model internalises the corrective effect without ever seeing weak-model patches at inference time.
This is a form of KL-regularised distributional update. The paper provides a conservative theoretical framing: the resulting token-level updates push the model toward continuations more associated with successful interventions, without claiming causal guarantees about final answer correctness.
Results and Ablations
Under greedy decoding across five benchmarks, Woodpecker Distillation improves average accuracy from 53.4% to 54.8% over the base strong model. The gains are largest on the hardest problems: AIME 2024 improves by 10 percentage points and AIME 2025 by 13.3 points. The method also outperforms self-rejection SFT trained on the same data budget, which filters and trains on the strong model's own correct self-generated trajectories.
Two ablations are worth noting. First, replacing the weak probe model with the strong model itself (same-model probing) drops average accuracy back to 53.4%, matching the baseline. This suggests the asymmetry between the probe and the target model is doing real work, not just the contrastive training mechanism in isolation. Second, swapping Gemma-3-4B-IT for Mistral-7B-Instruct-v0.3 as the probe yields stable results, suggesting the method is not sensitive to a specific weak model choice.
On compute, the dominant cost is verifier-based evaluation of patched continuations, which accounts for 86% of per-step runtime. Weak-model patch generation is under 1% of total cost. This is a practically important point: the bottleneck is verification, not the weak model itself, so scaling the method depends primarily on the cost of outcome evaluation rather than on running the probe.
Limitations and Open Questions
The authors are candid about several constraints. The evaluation is limited to mathematical reasoning, where automatic verification is straightforward. Whether localized reasoning bugs of this type appear with similar frequency in code generation, factual question answering, or multi-step planning is an open question. The method is also most natural in settings with reliable verifiers; extending it to domains where correctness is harder to assess automatically would require additional work.
There is also a scope limitation inherent to the framing. Woodpecker Distillation targets recoverable local bugs. If a model genuinely lacks the capability to solve a problem, or if the failure stems from a global planning error rather than a locally fixable misstep, the method has little to offer. The distinction between recoverable and unrecoverable failures is not always easy to determine in advance, and the paper does not provide a principled way to identify which failures fall into which category.
The verification cost bottleneck also raises scalability questions. At 86% of per-step runtime, reducing that cost would substantially change the economics of the method. Whether approximate or learned verifiers could substitute for exact outcome evaluation without degrading the contrastive signal is worth investigating.
More broadly, the finding that weak models can diagnose bugs in stronger models without being stronger end-to-end reasoners is an interesting data point for the weak-to-strong supervision literature. It suggests that the relationship between model capability and supervisory usefulness is more nuanced than a simple capability hierarchy would imply. A weak model that cannot solve a problem may still be able to generate interventions that reveal something structurally important about where a stronger model's reasoning goes wrong.