When a retrieval-augmented generation system hands a language model an answer from a database, or a tool returns a calculation result, what actually happens inside the model? The naive assumption is that the model weighs the external evidence against its internal knowledge and picks the more reliable one. A new paper, Evidence Integration in Large Language Models, argues this picture is fundamentally wrong, and backs that argument with over ten million experimental trials across twelve models and eight domains.
The core problem is that existing accounts of tool-augmented and retrieval-augmented LLMs treat evidence uptake as a function of source trust: how much does the model believe the tool? This paper shows that framing misses most of the variance. Whether a model accepts an externally supplied candidate answer depends not on a scalar trust score but on the model's own pre-existing distribution over answers, the structural relationship between the proposed candidate and the model's characteristic errors, and where in the network the integration decision is implemented. These are receiver properties, not source properties.
Key Contributions
The paper introduces a distributional theory of evidence integration. Before any external evidence arrives, the model holds a distribution q over candidate answers. Evidence proposing a specific candidate v transforms that distribution multiplicatively: the posterior is proportional to q raised to a prior weight (1+a), tilted by an evidence factor local to v. The prior weight parameter a is measured empirically and consistently falls between 0.20 and 0.65 across models and domains, well below the rational value of 1.0. This means models systematically underweight their own prior relative to a Bayesian ideal, but the degree of underweighting is model-domain specific rather than universal.
Three predictions follow from this theory, all confirmed experimentally:
- Candidates already well-supported by the model's internal distribution are more likely to be adopted when proposed externally.
- Models integrate their own characteristic errors more readily than structurally foreign errors from different sources, because the former already have higher prior support.
- Identical external evidence can improve performance on problems where the model is weak and degrade it on problems where the model is already competent, since the evidence shifts probability mass toward the proposed candidate regardless of whether that candidate is correct.
The paper also introduces a receiver-relative reliability frontier: the performance cost of receiver-congruent errors (errors the model was already inclined to make) is steeper than the cost of random errors at the same rate. This has direct implications for adversarial attacks and for understanding correlated failure modes in multi-agent systems.
Methodology
The experimental design is unusually careful about separating three quantities that most prior work conflates: generation competence G (probability of correct answer without any external candidate), checking competence C (probability of correctly judging a supplied candidate), and use probability U (probability of adopting the candidate). These are measured independently across eight domains, including four scientific reasoning tasks covering quantum mechanics, physics, genetics, and molecular biology, alongside arithmetic, word problems, linear systems, and propositional constraint satisfaction.
A cross-fit design separates item selection from analysis to avoid regression-to-the-mean artifacts at window boundaries. The arbitration law is estimated as a mixed-effects model decomposing policy variance into shared, model-level, domain-level, and interaction components. The interaction level accounts for 49 to 80 percent of between-cell variance depending on the term, which is the paper's clearest quantitative statement that evidence integration is a model-domain pair property rather than a model property alone.
The mechanistic component uses causal interventions and a J-lens decomposition to identify where in the network candidate integration is implemented. Knockout experiments suppress specific residual stream bands; dose-response steering experiments measure how much a certificate-aligned direction in representation space shifts the decision margin. These experiments cover seven models, with the two 70B checkpoints excluded due to memory constraints.
Main Findings
Several results stand out as particularly consequential. First, models accept externally proposed candidates even after internally verifying them as invalid at rates of 93 to 100 percent on propositional constraint tasks and up to 99.4 percent on held-out scientific reasoning. Verification and use are causally dissociated: the representation underlying a verbalized verification verdict can be decoded from the model's internal state but has little to no causal impact on the final answer. The J-lens decomposition shows these two representations are fully separable in the models tested.
Second, candidate integration is implemented late in the network as a structured sequence: admitting the external candidate, promoting it in the representation, and transporting that state toward the answer. This is consistent with earlier mechanistic interpretability work on how factual associations are stored and retrieved, but the paper extends that picture to the dynamic case where an external candidate competes with internally retrieved knowledge.
Third, the tool-source premium (the extra uptake probability when evidence is attributed to a tool rather than a user) is largest precisely where candidate support is weakest. This is a rational-looking pattern but it emerges from the integration law rather than from any explicit reliability assessment. The premium is also model-family dependent: positive for Llama and Gemma models in arithmetic, but it shifts sign once harder domains enter the fit.
Scale does not determine integration policy. The two same-scale Llama checkpoints (3.1-70B and 3.3-70B) differ substantially at fixed domain, and the 70B checkpoint that is nearly balanced on word problems adopts invalid SAT assignments at ceiling rates. Parameter count fixes neither the model-level effect nor the model-domain interaction.
Limitations and Implications
The authors are transparent about scope. The mechanistic interventions cover fewer models than the behavioral study. The reliability frontier is estimated only for the candidate distributions studied, and the prior weight is identified from answer-slot token probabilities, so it is measured where those probabilities are accessible. The composition laws for multiple sequential evidence messages are tested only on the arithmetic instrument. The scientific domain experiments use synthetic tasks with exact oracles, which allows clean separation of generation, checking, and use but leaves open how badly the same failure modes manifest with real retrieval systems, noisy measurements, or human reports.
The implications for system design are substantial. Scalar tool trust, the standard abstraction in most RAG and agent frameworks, cannot capture the structure this paper documents. A model can simultaneously accept a value it already favors from a low-reliability source, resist an equally wrong alternative from a high-reliability source, and fail to act on a check it can correctly perform. Designing around this requires knowing the receiver's answer distribution before evidence arrives, which is not currently a standard part of RAG pipeline design.
For AI safety and multi-agent systems, the finding that characteristic errors propagate preferentially is particularly important. If one model in a pipeline produces an error that falls within the characteristic error distribution of a downstream model, that error will be integrated more readily than a structurally foreign error of the same magnitude. Correlated errors across model families could therefore produce failure cascades that are much harder to detect than independent random errors at the same rate.
The full paper is available at arXiv:2609.04290.