Sander Dieleman's recent post at sander.ai offers one of the more careful technical retrospectives on continuous diffusion language models (CDLMs) that has appeared in the literature, and it arrives at a genuinely interesting moment. After a near-total extinction of continuous approaches following 2023, the space is seeing coordinated revival activity across multiple research groups simultaneously. Understanding why this happened, and why the reversal is occurring now, requires unpacking several interacting technical and sociological factors.
The Architecture of a Near-Extinction
The core tension between continuous diffusion and language modelling is not mysterious: language tokens are categorical, and Gaussian noise corruption acts on continuous spaces. The early CDLM literature from 2022, including Diffusion-LM, CDCD, and SED, addressed this mismatch by lifting the corruption process into a continuous embedding space. Rather than corrupting tokens directly (as masked or uniform-state discrete diffusion does), you embed them first and then apply Gaussian noise. This is clean, theoretically appealing, and opens access to the full toolkit developed for continuous audiovisual diffusion.
The extinction event that followed is well-documented in a 2025 survey Dieleman cites, which shows an almost binary transition from continuous to discrete methods between 2023 and 2024. The proximate cause appears to be the Gulrajani and Hashimoto (2023) result showing that their continuous model Plaid-1B was 64x less training-efficient than an autoregressive baseline on a likelihood basis. At a moment when the field was heavily Chinchilla-brained, that figure was effectively disqualifying. The broader community had not yet absorbed the LLaMA argument that inference-time compute should factor into the efficiency calculus, so continuous methods were evaluated purely on a training-compute-per-perplexity basis, and they lost badly.
This is worth sitting with for a moment, because it illustrates how strongly the evaluation criteria of a field can shape which research directions survive. Continuous diffusion was not shown to be theoretically inferior; it was shown to be less efficient under a specific metric that the community happened to prioritise at that moment. Had the inference-budget framing arrived six months earlier, the trajectory might have been quite different.
The Four Technical Ingredients and Their Difficulties
Dieleman's post is most valuable in its careful decomposition of what makes CDLMs technically hard to get right. Four components stand out:
- Embedding strategy: Jointly learned embeddings are the most attractive option from an end-to-end training perspective, but naive formulations are pathologically unstable. A denoiser can minimise reconstruction loss by collapsing all embeddings to a single point, so constraints or regularisation terms are necessary. The CDCD paper addressed this by normalising embeddings to the unit sphere, which is a reasonable but somewhat ad hoc solution.
- Loss function: The choice between MSE in embedding space and cross-entropy over vocabulary elements has significant downstream consequences. Cross-entropy requires that the denoiser produce per-token probability distributions, which constrains the architecture and is only tractable for moderate vocabulary sizes. MSE is more flexible but decouples the training signal from the discrete structure of the output.
- Noise schedule: This is perhaps the most underappreciated difficulty. For high-dimensional embeddings representing categorical data, the informative noise range is extremely narrow. Most noise levels either leave token identity essentially intact or destroy it completely, leaving almost nothing for the denoiser to learn. Adaptive schedules that linearise prediction entropy over diffusion time are a principled fix, but they add training complexity and create feedback loops that are nontrivial to stabilise.
- Self-conditioning: The empirical impact of self-conditioning on CDLM performance is striking and still not fully explained. The basic idea, passing the denoiser's previous prediction as an additional input to the next step, breaks the statelessness assumption underlying ODE/SDE-based sampling algorithms, yet the performance gains are large enough that essentially every competitive CDLM uses it anyway. The recent reanalysis by Yoo et al., framing self-conditioned diffusion as an approximation of a nested fixed-point loop, provides some theoretical grounding, but the full picture remains unclear.
The interaction between these four components is where most of the practical difficulty lies. Getting any one of them wrong tends to produce catastrophic failure modes rather than graceful degradation.
The 2025 Hybrid Phase and the Temporal Dissonance Problem
Before the full continuous revival of 2026, the intermediate period produced a cluster of hybrid methods that are worth examining carefully. CADD, CCDD, and CANDI each represent a different hypothesis about where discrete methods are weakest and how continuous representations can compensate.
The most technically interesting of these is the CANDI framing of temporal dissonance. The argument is that for large vocabularies, discrete token identity decays much faster during the corruption process than the relative ranking of tokens in embedding space. By the time the embedding geometry contains learnable semantic structure, individual token identities are already irrecoverable. This creates a mismatch between the timescales at which different kinds of information are destroyed, and it means the model cannot simultaneously learn fine-grained token identity and coarse-grained semantic structure from the same corruption trajectory.
This is a genuinely important observation, and it has a direct parallel in the image diffusion literature: the well-known phenomenon that coarse structure (composition, colour palette) is determined at high noise levels while fine details (textures, edges) are resolved at low noise levels. For language, the analogous decomposition between token identity and semantic structure is not naturally aligned with a single Gaussian corruption process, which is exactly what CANDI identifies. Their solution, using discrete masking to handle token identity and Gaussian noise only for masked positions, is essentially a way of decoupling these two timescales by using two different corruption mechanisms.
Flow Maps and the 2026 Resurgence
The 2026 revival of purely continuous methods appears to be substantially driven by the development of flow maps, which Dieleman covered in his previous post. A flow map approximates the entire trajectory of a diffusion sampling procedure in a single forward pass, or at least in far fewer steps than standard diffusion sampling requires. The appeal for language is clear: one of the persistent criticisms of diffusion-based language generation is inference speed relative to autoregressive models, and flow maps directly attack that bottleneck.
The appearance of multiple closely related CDLM papers in early 2026 suggests that several groups reached similar conclusions about the viability of continuous methods at roughly the same time. This kind of simultaneous convergence is usually a signal that some underlying technical barrier has been cleared, rather than coincidence. The most plausible candidate is the combination of flow map machinery with the now-mature toolkit of adaptive noise schedules and self-conditioning, which together address the three main failure modes that plagued 2022-era CDLMs.
There is also a subtler factor worth noting. The framing of score interpolation from the CDCD paper, which enabled cross-entropy-based training of continuous denoisers, has apparently been rederived in the flow matching framework with stronger theoretical backing. This matters because it means the connection between continuous diffusion and the familiar categorical cross-entropy loss is now on firmer ground, making CDLMs more interpretable to practitioners trained on autoregressive models.
What This Revival Actually Implies
The honest assessment is that CDLMs are not yet competitive with the best discrete diffusion models at scale, and neither class is yet competitive with state-of-the-art autoregressive models on most benchmarks. The revival is driven by theoretical promise and a set of converging technical improvements, not by demonstrated empirical superiority.
That said, the theoretical advantages Dieleman identifies are real. The ability to represent uncertainty at the individual token level, rather than making hard discrete decisions at each corruption step, is genuinely useful for tasks that require nuanced probabilistic reasoning. The richer sampling toolkit available for continuous methods, including higher-order ODE solvers and guidance mechanisms developed for image and video generation, represents a substantial accumulated resource that discrete methods cannot easily access.
The more interesting question for the next two to three years is whether the scaling behaviour of CDLMs will prove fundamentally different from discrete methods, or whether the 64x efficiency gap identified in 2023 can be substantially closed through architectural and training improvements. The adaptive schedule and flow map developments suggest meaningful progress is possible. Whether it is sufficient to make CDLMs the preferred approach at the frontier, rather than a theoretically interesting alternative, remains genuinely open.
For practitioners, the practical takeaway is that the CDLM toolkit is now mature enough to be worth serious evaluation for constrained generation tasks, infilling, and applications where token-level uncertainty quantification is valuable. The extinction event of 2023 was a product of specific evaluation criteria at a specific moment, not a verdict on the fundamental viability of the approach.