The computational cost of processing very long sequences has been one of the most stubborn constraints on large language model development. Vanilla attention scales quadratically with sequence length, which makes million-token contexts prohibitively expensive at inference time and severely limits how far test-time scaling can go before hitting a wall. DeepSeek-V4 is a direct attempt to break that constraint through architectural redesign rather than hardware brute force.
The paper presents two Mixture-of-Experts models: DeepSeek-V4-Pro (1.6T total parameters, 49B activated) and DeepSeek-V4-Flash (284B total, 13B activated). Both natively support one million token contexts. The headline efficiency claim is striking: in the 1M-token setting, DeepSeek-V4-Pro requires only 27% of the single-token inference FLOPs and 10% of the KV cache of its predecessor DeepSeek-V3.2. DeepSeek-V4-Flash pushes this further to 10% of FLOPs and 7% of KV cache. These are not marginal gains from quantisation tweaks; they reflect genuine architectural changes.
Key Contributions
Three architectural innovations sit at the core of DeepSeek-V4, layered on top of the existing DeepSeekMoE and Multi-Token Prediction framework inherited from V3:
- Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA): CSA compresses KV caches along the sequence dimension before applying DeepSeek Sparse Attention. HCA applies more aggressive compression but retains dense attention. These two mechanisms are combined in a hybrid attention architecture, with different layers using different attention types depending on efficiency requirements. The net effect is a dramatic reduction in both KV cache footprint and attention FLOPs, particularly at long sequence lengths.
- Manifold-Constrained Hyper-Connections (mHC): This replaces conventional residual connections with a geometrically constrained variant that is claimed to improve gradient flow and modelling capacity. The authors treat this as an upgrade to the foundational connectivity of the transformer rather than a module-level addition.
- Muon Optimizer: The training uses a momentum-based optimiser with a Newton-Schulz orthogonalisation step applied to gradient matrices. This is reported to improve convergence speed and training stability relative to AdamW, which is a meaningful claim given the scale involved. The paper provides the algorithm pseudocode and describes a hybrid ZeRO strategy to make distributed Muon training tractable.
On the storage side, KV entries use a mixed precision format: BF16 for RoPE dimensions and FP8 for remaining dimensions, cutting KV cache size roughly in half compared to pure BF16. Attention computation within the lightning indexer runs in FP4. The routed expert weights also use FP4, with a note that current hardware cannot yet exploit the theoretical throughput advantage of FP4 operations, but future hardware should.
Training and Post-Training Pipeline
Both models are pre-trained on over 32 trillion tokens. The post-training pipeline is a two-stage process that is worth examining carefully because it departs from the standard SFT-then-RLHF recipe in a few ways.
First, domain-specific expert models are trained independently for mathematics, coding, agent tasks, and instruction following. Each expert undergoes SFT on domain-specific data followed by GRPO-based reinforcement learning with domain-tailored reward models. Second, a unified model is trained via on-policy distillation from this ensemble of specialists, with the student optimising a reverse KL loss against the teacher outputs.
For hard-to-verify tasks where scalar reward models are traditionally used, the authors instead train a Generative Reward Model (GRM) in which the actor network simultaneously functions as the evaluator. This eliminates the need for large-scale human annotation of reward signals and reportedly produces more calibrated scoring by grounding evaluation in the model's own reasoning traces.
Results
The base model comparisons in Table 1 are informative. DeepSeek-V4-Flash-Base, with only 13B activated parameters, outperforms DeepSeek-V3.2-Base (37B activated) on the majority of benchmarks, including long-context tasks (LongBench-V2: 44.7 vs 40.2) and several knowledge benchmarks. DeepSeek-V4-Pro-Base then extends this further across nearly all categories, with particularly large gains on knowledge-intensive benchmarks like FACTS Parametric (62.6 vs 27.1) and Simple-QA Verified (55.2 vs 28.3).
For the instruction-tuned models, the key comparisons from Table 6 are:
- LiveCodeBench: DeepSeek-V4-Pro-Max scores 93.5, above Gemini-3.1-Pro (91.7) and Kimi-K2.6 (89.6)
- Codeforces rating: 3206, the highest reported in the comparison set
- Long-context MRCR at 1M tokens: 83.5, above Gemini-3.1-Pro (76.3) and Opus-4.6 (92.9), though below the latter
- SWE-Verified: 80.6, on par with Opus-4.6 and Gemini-3.1-Pro
The authors are candid that DeepSeek-V4-Pro-Max trails GPT-5.4 and Gemini-3.1-Pro on several reasoning benchmarks, estimating a development gap of roughly three to six months relative to frontier proprietary models. This is an unusually direct self-assessment and lends credibility to the reported numbers.
DeepSeek-V4-Flash-Max is positioned as a cost-efficient alternative that approaches GPT-5.2 and Gemini-3.0-Pro performance on reasoning tasks when given a larger thinking budget, which is a useful framing for deployment scenarios where compute cost per token matters more than absolute capability.
Limitations and Implications
The authors acknowledge that the architecture is complex, partly because they retained many previously validated components to reduce risk during development. This is an honest engineering admission: the hybrid attention scheme, mHC connections, MoE routing, MTP modules, and mixed-precision KV storage interact in ways that are difficult to analyse cleanly. Future iterations aim to simplify. The training instability mitigations (Anticipatory Routing, SwiGLU Clamping) are noted to work empirically without a well-understood theoretical basis, which is a gap worth watching.
The FP4 expert weight claim deserves scrutiny. Current hardware does not provide a throughput advantage for FP4 over FP8 matrix operations, so the efficiency gains cited in the paper are based on existing hardware capabilities. The theoretical 1/3 improvement is contingent on hardware that does not yet exist at scale.
The broader implication is significant. If million-token contexts can be made computationally tractable at this level of efficiency, the bottleneck on test-time scaling shifts from context length to reasoning quality and training data. This opens up genuinely new application territory: cross-document reasoning, long-horizon agentic tasks, and potentially online learning where the model processes extended interaction histories. The open release of model weights also matters here; the efficiency claims become practically testable by the research community rather than remaining proprietary benchmarks.
The Muon optimizer adoption at this scale is also worth tracking. Its use in large MoE training is relatively novel and the convergence improvements, if reproducible, could influence how future large models are trained independent of the specific architectural choices in DeepSeek-V4.
Full paper: arXiv:2606.19348. Model weights are available via the DeepSeek HuggingFace collection.