Chain-of-thought prompting has become a standard technique for improving arithmetic performance in language models, yet a persistent puzzle has gone largely unaddressed: if you shuffle the steps of a CoT solution into a random order, accuracy barely drops. This suggests the model is not actually following the logical sequence it appears to be using. A new paper, The Readout Shortcut: Positional Number Copying Dominates Arithmetic CoT Readout in Small Language Models, provides a concrete mechanistic answer for the 1-3B parameter regime: these models are predominantly copying whichever number appears at the trailing position before the answer delimiter, not reasoning through intermediate steps.
This matters well beyond a narrow curiosity about small models. Step-level faithfulness evaluations are increasingly used as proxies for whether a model is genuinely reasoning. If what looks like a correct answer is often just positional copying, those evaluations are measuring the wrong thing, with real consequences for CoT-based oversight and interpretability work.
What the Paper Contributes
The study operates on three instruction-tuned models spanning distinct architectures: Qwen2.5-1.5B-Instruct, Llama-3.2-1B-Instruct, and Gemma-2-2B-it. Using GSM8K as the primary benchmark, it identifies what the authors call an answer-context-gated positional readout: the model reads the number at the trailing position in answer-relevant context before the #### delimiter, largely regardless of what the intermediate reasoning steps contain.
Four converging lines of evidence support this claim:
- A corruption decomposition showing that gold-answer presence at the trailing position accounts for 54-92 percentage points of raw accuracy (89-92% of each model's teacher-forcing ceiling).
- A distractor injection experiment demonstrating that Qwen and Llama copy a novel wrong number 87-95% of the time when it is placed at the trailing position, even when the correct answer appears in the intermediate steps.
- A position sweep showing a sharp final-slot jump of 20-31 pp, confirming the effect is specifically tied to the last position before the delimiter rather than a smooth recency gradient.
- Head-level ablations revealing architecture-specific profiles, ranging from localised (Llama) to distributed (Qwen).
A particularly important finding is the causal ordering. The copy channel does not merely compete with retained-context computation; it suppresses it. When a trailing number is present, even single-step arithmetic the model could otherwise perform is overridden. Removing the trailing number recovers 5-32 pp above the near-zero floor caused by an injected wrong number, confirming that intermediate steps carry some exploitable signal that the positional shortcut actively masks.
Methodology and Controls
The experimental design is careful about alternative explanations. The paper explicitly maps ten competing hypotheses to dedicated controls, including gold-specific semantic retrieval, smooth recency bias, answer-template parsing, delimiter surface-form artifacts, GSM8K memorisation, and instruction-tuning artifacts. Each is addressed by at least one targeted experiment.
The framing-dissociation experiment is particularly clean. Placing a bare number at the trailing position yields low copying rates (0.45 for Qwen, 0.26 for Llama), while wrapping the same number in answer-relevant inline framing recovers high copying rates (0.77 and 0.85 respectively). Same number, same position, but copying probability roughly doubles based purely on surrounding text framing. This dissociates the mechanism from both raw positional recency and template-specific parsing.
The base-model probe confirms the mechanism is present in pre-instruction-tuning weights. Instruction-tuning amplifies copy strength but does not introduce it. The effect also replicates on GSM-Symbolic, ruling out benchmark memorisation as an explanation.
One methodological caveat worth flagging: Gemma's distractor experiments (Experiments 3 and 4) required restriction to a teacher-forcing-passing subset due to fidelity drops under prefix-structure perturbation. On that subset, Gemma's behaviour is qualitatively different from Qwen and Llama, showing much stronger content gating (P(distractor) = 0.12-0.19 versus 0.87-0.95 for the other two models). The paper handles this transparently, conditioning Gemma's results appropriately and noting where it diverges.
Key Findings and Architectural Variation
The results decompose cleanly into two independently varying components. Gold-answer dependence at the trailing position is consistent across all architectures and scales tested. The content gate, meaning the degree to which the model rejects a novel distractor, varies substantially:
- Qwen (1.5B): content gate approximately zero; copies novel distractors ~95% of the time.
- Llama (1B): partial content gate; copies novel distractors ~87-95% of the time but shows tighter numeric-target selection.
- Gemma (2B): strong content gate (~0.85); selectively resists distractors.
- 7-8B models: content-selective gating emerges consistently (gate >= 0.70).
The scale dependence is important. At 7-8B parameters, the pattern shifts toward genuine content-selective retrieval, suggesting the positional shortcut is not a universal property of transformer-based language models but a characteristic of the low-parameter regime. Non-arithmetic BBH tasks also show sharp drops in shuffle-retention, indicating the shortcut is specific to arithmetic rather than a general property of CoT in these models.
The finding that DeepSeek-R1-Distill-Qwen-1.5B retains the shortcut despite longer reasoning traces (ceiling-normalised delta_copy = 0.928) is a useful data point. Reasoning training partially engages the content gate (P(distractor) drops to 0.706 from 0.87-0.95) but does not remove the positional mechanism. Longer rationales do not fix the underlying readout behaviour.
Implications and Open Questions
The most direct implication is for CoT faithfulness evaluation. Step-level metrics that treat a correct final answer as evidence of genuine step-by-step computation are conflating positional answer transport with reasoning. This is a meaningful failure mode for interpretability work and for any oversight framework that relies on CoT transparency as a signal of model behaviour.
The paper also reframes why shuffling CoT steps preserves performance. It is not that the model extracts some diffuse logical signal from disordered steps; it is that the answer token's positional accessibility survives step reordering. The readout mechanism does not care about logical sequence because it was never using logical sequence in the first place.
Several questions remain open. The distractor and position effects under unconstrained (non-prefix-completion) generation are acknowledged as future work. The head-level ablations identify sensitivity profiles rather than complete circuits, so a full mechanistic account of how the copy channel is implemented remains incomplete. It is also unclear how the transition from positional copying to content-selective retrieval occurs as model scale increases, and whether targeted training interventions could shift the content gate without degrading arithmetic performance.
For practitioners deploying small language models in arithmetic-heavy applications, the implication is straightforward: CoT formatting that places intermediate results at the trailing position before the answer delimiter is likely being exploited as a copy cue rather than a reasoning scaffold. Whether that matters depends on whether the final answers are correct, but it does mean that apparent CoT reasoning in this regime should not be taken as evidence that the model is doing what it appears to be doing.
The paper is available in full at arXiv:2605.22870.