Most work on improving large language model outputs focuses on training: better data, better objectives, better fine-tuning. A quieter but increasingly important thread asks a different question: given a fixed model, can we do better at inference time by sampling more intelligently? This paper, available at arXiv:2608.26120, takes that question seriously and provides a principled answer grounded in statistical inference theory.
The core problem is that LLMs are probabilistic models, but the sampling strategies used in practice are unsophisticated relative to what the probabilistic inference literature has developed over decades. Standard decoding, temperature scaling, and Best-of-N are effective but theoretically ad hoc. This paper attempts to close the gap between LLM inference practice and the richer toolkit of Monte Carlo methods.
Key Contributions
The paper makes three main contributions that build on each other:
- A unified target distribution framework. The authors define a class of flexible target distributions over sequences, including powered distributions (sharpening the base model), product-form distributions (combining multiple signals), and tilted distributions (reweighting by some auxiliary function). These generalize what practitioners already do informally with temperature and reranking.
- A transport-based view of autoregressive generation. By treating the token-by-token generation process as a time-dependent transport from an empty sequence to a complete one, the authors establish a generalized Bayes' rule for LLMs. This reframing is what allows principled sequential Monte Carlo methods to be applied.
- Two concrete algorithms: SMC and Replica Exchange. Sequential Monte Carlo maintains a population of partial sequences with importance weights, resampling at intermediate steps. Replica Exchange runs multiple chains at different temperatures and proposes swaps between them. Both are adapted carefully to the discrete, variable-length structure of autoregressive generation.
The entropy tilting approach is worth highlighting separately. Rather than relying on an external verifier or reward model, the authors tilt the target distribution using the model's own predictive entropy. Chunks where the model is more confident (lower entropy) are upweighted. This is an intrinsic signal that requires no additional supervision, which matters practically since reward models are expensive and domain-specific.
Methodology
The generation process is chunked into segments of up to 192 tokens, with 16 chunks per sequence and a maximum generation length of 3072 tokens. The proposal distribution is a low-temperature autoregressive sampler (temperature 0.25), while the target is a powered version of the base distribution with power parameter alpha equal to 4. The entropy tilt coefficient lambda is swept over {0, 100, 500, 1000}.
This chunked structure is important. Full-sequence importance weighting would collapse catastrophically in high dimensions; chunking provides intermediate resampling opportunities that keep the particle population from degenerating. The approach is analogous to what SMC does in state-space models, adapted here to the discrete sequential structure of text.
Experiments run on MATH500 and GPQA-Diamond, using Qwen2.5-Math-7B, Qwen2.5-7B, and Qwen2.5-32B-Instruct. The choice to use GPQA-Diamond with the 32B instruct model rather than the 7B base model is honest: the authors note the smaller model does not reliably follow the required answer format, which would confound evaluation. That kind of methodological transparency is reassuring.
Results and What They Mean
The headline finding is that both SMC and RE scale more favorably than standard MCMC baselines and outperform naive low-temperature sampling. The two algorithms show complementary strengths:
- SMC is better at producing a strong single answer, making it the better choice when you need one high-quality output.
- RE preserves sample diversity more effectively, leading to better pass@k performance when you care about coverage across multiple samples.
- Best-of-N remains a competitive baseline for single-answer selection, but SMC and RE sample from the full target distribution rather than just selecting the highest-scoring output from independent draws.
- Entropy tilting provides additional gains without any external verifier, suggesting that the model's own uncertainty is an underused signal.
The distinction between SMC and RE in terms of diversity versus quality is theoretically expected. SMC with resampling tends to collapse diversity over time as low-weight particles are eliminated. RE, by maintaining chains at multiple temperatures and allowing swaps, retains more of the distribution's spread. Seeing this play out empirically on reasoning benchmarks is useful confirmation.
Limitations and Open Questions
The authors are candid about the computational overhead. Both SMC and RE require running the model multiple times per output, which is more expensive than standard decoding. For applications where latency matters or compute is constrained, Best-of-N may still be the practical choice despite being theoretically less principled.
There is also a dependency on base model quality. The framework improves inference given a capable model, but a weak base model will saturate quickly. This is not a criticism of the paper so much as a structural constraint of inference-time methods generally.
A few open questions are worth thinking about. First, the chunking strategy is somewhat heuristic: 16 chunks of up to 192 tokens is a reasonable choice, but the sensitivity to chunk boundaries and sizes is not fully characterized. Second, the entropy tilt is applied uniformly across chunk positions, but there may be structure in when uncertainty is informative versus noise. Third, the experiments focus on mathematical reasoning, where correctness is binary and verifiable. How well these methods transfer to tasks with softer quality criteria is an open question.
The connection to diffusion model inference-time control is noted in the paper and is genuinely illuminating. Methods like Feynman-Kac guidance and annealed importance sampling have been developed extensively for continuous generative models. Adapting them to discrete, variable-length autoregressive generation is non-trivial, and this paper makes a credible contribution to that translation. The field is likely to see more cross-pollination between diffusion model inference methods and LLM decoding as both communities recognize the shared probabilistic structure.
Overall, this is a technically solid paper that advances the principled treatment of LLM inference as a sampling problem. It will be most immediately useful to researchers working on inference-time scaling, controlled generation, and the theoretical foundations of LLM decoding. The full paper is available at https://arxiv.org/abs/2608.26120.