Most supervised fine-tuning pipelines for code language models apply cross-entropy loss uniformly across every token in a response. This is a convenient default, but it rests on a questionable assumption: that every token in a generated code response carries equally useful learning signal. A growing body of work on token-level selection in natural-language SFT has already challenged this assumption, but translating those ideas directly to code turns out to be more complicated than it appears. The paper CodeBlock: Learning to Supervise Code at the Right Granularity addresses exactly this gap, proposing a structure-aware sparse supervision framework that achieves competitive or superior performance to full-token SFT while supervising fewer than 2% of response tokens.
The Core Problem: Granularity Mismatch in Code Supervision
Token-level selection methods in NLP typically score individual tokens by their pointwise cross-entropy or excess-loss and retain only the high-scoring ones. In natural language, this works reasonably well because individual tokens often carry interpretable local semantics. Code is different. A variable name, an operator, or a bracket in isolation does not constitute a meaningful unit of program behaviour. Meaning in code emerges from syntactically closed constructs: assignment statements, conditional branches, return expressions, API calls. Selecting scattered high-loss tokens without regard to these structural boundaries produces supervision targets that are syntactically incomplete and semantically incoherent.
The authors substantiate this claim with a careful diagnostic study. Sampling 30K code responses and scoring tokens with a frozen Qwen2.5-Coder-1.5B-Instruct model, they find that even at a top-10% token budget, 92.3% of the syntactic items touched by selected tokens are only partially covered. Recovering syntactically closed fragments from those selections requires a 7.02x expansion in token count. A controlled mini-SFT experiment then shows that supervising local syntactic closures rather than isolated high-CE tokens improves average benchmark performance by 11.7 points under the same token budget. This is a compelling empirical motivation, not just a theoretical concern.
Methodology: Structure-Aware Sparse Supervision
CodeBlock operates in four stages:
- Sample-level selection. A standard quality filter selects 30K high-quality instruction-response pairs from a 300K pool drawn from OpenCodeInstruct. This is largely conventional and not the main contribution.
- Coding item partitioning. Code regions in each response are segmented into minimal syntactically coherent units, called coding items. Each item has a set of core logic tokens (identifiers, literals, operators, API names) and a materialized closed fragment that includes the boundary tokens needed for syntactic completeness.
- GCE-based utility scoring. Each coding item is scored by aggregating generalised cross-entropy over its core logic tokens. GCE is used rather than standard CE because it is less sensitive to very high-loss outliers that may reflect noise rather than genuine informativeness.
- Data-flow-guided reranking. Items are reranked using two lightweight static analysis signals: a reach score measuring how many downstream tokens depend on a given item's definitions, and a bridge score measuring whether the item lies on dependency paths connecting other high-utility items. This prioritises blocks that propagate or connect important program dependencies.
During training, the full response remains in the autoregressive context, so the model sees the complete program. Loss is applied only to selected code items and informative natural-language tokens. This decoupling of contextual exposure from gradient supervision is a clean design choice: the model learns from structurally meaningful targets without losing the coherence that comes from seeing complete responses.
Results Across Six Benchmarks
Experiments span HumanEval, HumanEval+, MBPP, MBPP+, BigCodeBench-Hard, and BigCodeBench-Full, evaluated across five model configurations: Qwen2.5-Coder at 1.5B, 3B, and 7B, Seed-Coder-8B, and OpenCoder-8B-Base. Key findings:
- CodeBlock consistently matches or outperforms full-token SFT with approximately 1.9% effective supervised tokens across all model sizes and initialisations.
- On Qwen2.5-Coder-1.5B-Instruct, average pass@1 improves by 1.9 points over full-token SFT. On OpenCoder-8B-Base, the improvement is 2.6 points.
- Token Cleaning, a strong domain-agnostic token selection baseline, underperforms the base model on several benchmarks, reinforcing the argument that code-specific structural constraints matter.
- At 7B scale, CodeBlock reaches an average of 65.4 versus 64.4 for full-token SFT and 62.2 for Token Cleaning.
- Wall-clock preprocessing cost is around 32 minutes, comparable to other selection methods, with training time similar to sparse baselines at roughly 9 minutes versus 246 minutes for full-token SFT.
The efficiency gains in training time are substantial. Full-token SFT on this setup takes around 246 minutes of training; CodeBlock reduces that to roughly 9 minutes of SFT after 32 minutes of preprocessing, a total of 41.5 minutes versus 255 minutes. That is a 6x reduction in total compute for equal or better downstream performance.
Limitations and Open Questions
The authors are candid about the limitations of their data-flow analysis. The def-use graphs are constructed via static analysis, which cannot capture runtime-dependent behaviours: dynamic dispatch, aliasing, object mutation, or input-dependent execution paths. Reach and bridge scores are therefore approximate structural signals rather than precise program semantics. Whether more precise analysis, such as incorporating execution traces or abstract interpretation, would further improve selection quality is an open question.
There is also a question of language coverage. The experiments focus on Python-centric benchmarks. Code structure varies considerably across languages; the syntactic partitioning and static analysis components would need adaptation for languages with different scoping rules, type systems, or evaluation strategies. The generalisation of the approach to multilingual code corpora is not evaluated.
A subtler issue is that the paper evaluates on functional correctness benchmarks, which measure whether generated code passes test cases. It is less clear how structure-aware supervision affects other desirable properties such as code readability, adherence to style conventions, or performance on security-sensitive tasks. The supervision signal selected by CodeBlock is biased toward computationally dense, logically central blocks; whether this introduces any systematic biases in generation style is worth investigating.
The broader implication is that domain-specific structure should inform supervision design, not just data filtering. The success of CodeBlock suggests that the field may have underinvested in understanding what constitutes a meaningful learning unit in different output modalities. The same logic could apply to mathematical reasoning, where expression-level or proof-step-level supervision might be more appropriate than token-level selection, or to structured generation tasks more generally. CodeBlock offers a concrete and well-motivated template for thinking about this problem in the code domain.
The full paper is available at arxiv.org/abs/2606.18286.