As large language models become embedded in high-stakes applications, the question of why a model produced a particular output is no longer academic. Most current explainability methods for LLMs either require access to model gradients, making them architecture-specific, or rely on perturbation-based surrogates that introduce arbitrary hyperparameter choices. A new paper from researchers at Argonne National Laboratory, Probabilistic Attribution For Large Language Models, takes a different route: grounding token attribution directly in the probability theory that LLMs already use during inference.
The core observation is straightforward but underexplored. LLMs are autoregressive stochastic processes. Every token they generate is sampled from a conditional probability distribution over the vocabulary, given all preceding tokens. Those distributions are not just a mechanism for decoding; they encode the full distributional structure the model learned during training. The authors ask whether those probabilities can be inverted, via Bayes rule, to say something meaningful about which prompt tokens were most responsible for a given response.
Key Contributions
The paper defines an attribution score (AS) as the log-ratio of two quantities: the conditional probability of the response given the full prompt, and the conditional probability of the response given the prompt with a specific token marginalized out. Tokens that, when removed from the marginal distribution, cause a large drop in response probability receive high attribution scores. Tokens whose removal changes little score near zero.
Several properties make this approach worth taking seriously:
- Model-agnostic: AS requires only the log-probabilities emitted during inference, not gradients or internal activations. It applies equally to GPT-2, Llama, Qwen, OLMo, or any other autoregressive model.
- Parameter-free: Unlike LIME, SHAP variants, or ReAgent, there are no user-defined hyperparameters. This removes a significant source of variability in XAI comparisons and makes AS a plausible baseline for calibrating other methods.
- Sensitivity by construction: Because AS is derived from first principles of probability theory, the score itself encodes how sensitive the model's output is to each token, rather than treating sensitivity as a separate diagnostic.
- Entropy measures for stability: Alongside AS, the authors define contextual entropy and replacement entropy, which quantify how concentrated or diffuse the model's token distributions are, conditioned on the surrounding context. These serve as indicators of model convergence and output stability.
Methodology
The mathematical machinery sits within the theory of stochastic processes. The joint probability of a token sequence factorises via the chain rule into a product of conditional probabilities, which is precisely how autoregressive LLMs generate text. The authors reconstruct the probability distribution over token sequences from the log-probabilities the model emits at inference time, without any access to training data or model weights beyond what standard inference APIs expose.
To compute AS for a given prompt token, the method marginalises that token out of the joint distribution and computes the resulting change in the log-probability of the response. In practice this requires summing over the full vocabulary as replacement candidates, which is computationally expensive. The authors run experiments on an A100 GPU at the Argonne Leadership Computing Facility and acknowledge that restricting summations to top-probability tokens is a natural optimisation for future work.
Experiments cover 8 models ranging from GPT-2 (137M parameters) to Qwen3-1.7B, across 7 prompts drawn from standard NLP benchmarks and modified to include confusing or deceptive constructions. Both greedy and nucleus (top-p) decoding are evaluated. For top-p sampling, the most frequent response across 500 generations is used as the fixed response for attribution analysis.
Results and Evaluation
AS is benchmarked against five established XAI methods: Input X Gradient, Gradient Shapley, LIME, Occlusion, and ReAgent. Four evaluation metrics are applied: Infidelity, Deletion/NAOPC curves, Comprehensiveness, and Sufficiency. Across most model-prompt combinations, AS performs comparably to or better than the competing methods, which is notable given that it requires no architecture access and no hyperparameter tuning.
The entropy analyses yield some practically useful findings. Near-zero AS values, combined with low contextual entropy, indicate tokens to which the model is largely indifferent, suggesting stable, well-converged behaviour in that region of the prompt. Large positive AS values flag tokens that are genuinely load-bearing for the response. Negative AS values, where marginalising a token actually increases response probability, point to tokens that may be misleading or confusing the model.
The comparison between older models (GPT-2, GPT-Neo) and newer instruction-tuned models (Llama-3.2, Qwen variants) is illuminating. The entropy and KL divergence measures suggest that GPT-2 and GPT-Neo are less well-converged on their training distributions relative to newer models, a conclusion drawn without any access to training data. The authors suggest this could eventually be used to monitor training convergence during LLM development, identifying bad checkpoints or insufficient model capacity from inference-time statistics alone.
Limitations and Implications
The most significant practical limitation is computational cost. Computing AS requires iterating over the full vocabulary for each prompt token, which scales poorly as vocabulary size grows. The paper reports total GPU hours that are substantially higher than gradient-based methods. The authors are candid about this and propose vocabulary pruning as a future optimisation, but until that work is done, AS is better suited to offline analysis than real-time deployment.
There is also a conceptual boundary the authors are careful to observe: AS captures text-distributional properties of inference, not anything about knowledge representation or internal reasoning circuits. This is methodologically honest but means AS cannot answer questions about why a model associates certain tokens with certain responses at a mechanistic level. For that, one would need to combine AS with mechanistic interpretability tools.
The evaluation relies on standard faithfulness metrics (Infidelity, Comprehensiveness, Sufficiency) that themselves have known limitations. These metrics measure consistency between attribution scores and model behaviour under token deletion, but deletion can introduce distribution shift. The authors use token replacement rather than masking to mitigate this, which is the more defensible choice, but the broader critique of perturbation-based evaluation still applies.
For the XAI field, the most interesting contribution may be the parameter-free property. A persistent problem in comparing explainability methods is that results depend heavily on arbitrary choices like perturbation budget, number of LIME samples, or ReAgent probe count. A theoretically grounded, parameter-free baseline could help standardise evaluation across methods, and the authors explicitly position AS for this role.
The code is available at github.com/sshilpika/probabilistic-attribution-score. For a field that often struggles to reproduce XAI results, open-source release alongside a parameter-free method is a meaningful practical contribution.