When a memory or retrieval-augmented generation (RAG) system answers a question, it does not pass the raw conversation to the model. It passes a rendering of that conversation: a summary, a structured record, a natural-language memory entry, or a truncated excerpt. A new paper, RENDER: Controlling Reader-Facing Evidence in LLM Memory Evaluation, makes the case that this rendering choice is not a neutral implementation detail. Across nine commercial models and roughly 238,000 model calls, measured accuracy on the same underlying questions swings by as much as 73 percentage points depending solely on what artifact reaches the answering model. That is a methodological problem for the entire field of memory and RAG evaluation.
The Core Problem
Memory benchmarks routinely compare systems that hand the reader different artifacts. One pipeline passes raw retrieved dialogue; another passes a structured memory record; another passes a compressed summary. If those artifacts are not reported or held constant, the final score conflates model reasoning ability, retrieval quality, memory-update logic, and evidence rendering into a single number. A reported accuracy gain could mean the model got better, or it could mean the rendered artifact got cleaner. The two are currently indistinguishable in most published evaluations.
The paper frames this around a concrete example: a user who says they live in Boston, then later says they moved to Denver, then asks "Where do I live now?" A ChatGPT-style memory entry might read "User lives in Denver (moved from Boston)"; a MemGPT-style typed record might expose user.city = "Denver"; a LangChain summary might compress the session into a paragraph. All three encode the same fact. None of them are equivalent reader inputs, and the experiments confirm they do not produce equivalent accuracy.
What RENDER Contributes
RENDER (Reader Evidence Rendering Diagnostics) is a benchmark control, not a new dataset. It fixes the underlying conversation, the question, and the answer contract, and varies only the reader-facing artifact. The framework has two instruments:
- A five-level packet ladder. Packets P0 and P1 expose only witness addresses (pointers to where answer-bearing content lives). P2 writes the resolved current-state value directly into the packet body. P3 and P4 add metadata around that value. This ladder localises failure: if a model fails at P2 but not P3, the problem is not missing content but something about metadata density or surface form.
- A template-family comparison. The same underlying dialogue is rendered as deterministic approximations of four deployed memory surfaces: ChatGPT-style natural-language entries, LangChain summaries, MemGPT-style typed records, and raw conversation. Templates are generated once per question at temperature zero and held fixed across all evaluator models, so reader behaviour is isolated from generator stochasticity.
The study runs on 500 questions from LongMemEval, with auxiliary experiments on LoCoMo and HotpotQA. All nine models (from OpenAI, Anthropic, and Google) have provider-native tools, search, and memory disabled. An independent LLM judge rescores the headline experiments, with a 600-item second-judge sample confirming inter-judge agreement.
Key Findings
The results are striking enough to warrant quoting directly from the paper:
- Budget-matched comparison: Truncating raw conversation to approximately 800 words (keeping the most recent turns) scores 7.6 to 12.0% across all nine models. A streamlined resolved P2 packet on the same budget scores 50.0 to 82.0%. The gap is 42.4 to 72.6 points, significant under paired bootstrap for every model. The judge-rescored gap is larger still, ranging from 48.4 to 80.2 points.
- Deployed-style templates: The best-to-worst accuracy spread across template families is 24.6 to 48.8 points per model. ChatGPT-style entries have higher point estimates than raw conversation on 7 of 9 models under the primary scorer, though judge rescoring shows significance on only 3 of 9 models and finds two models where ChatGPT-style entries are significantly worse than raw.
- Hard abstention: Three models score 0% on formal ledger-style packets with explicit abstention machinery. The same models answer the same underlying facts at 45.4 to 53.4% when those facts are wrapped in natural-language memory entries. This is not a content problem; it is an artifact-and-prompt interaction that would be invisible in any evaluation that does not report or control the reader-facing surface.
- Transfer: The effect holds under retrieval noise and transfers to HotpotQA, suggesting this is not an artefact of LongMemEval's structure.
Methodology: Strengths and Limitations
The study is carefully designed in several respects. Fixing the answer contract and varying only the input-side artifact is the right way to isolate rendering as a variable. The packet ladder is a genuinely useful diagnostic tool because it localises failures rather than just measuring them. Using a secondary LLM judge and reporting per-model confidence intervals with bootstrapping adds credibility, and the authors are transparent about where significance is and is not established.
There are real limitations worth considering. The primary accuracy metric is substring matching, which the authors acknowledge is imperfect; the judge-rescored results are more semantically grounded but introduce their own reliability questions. The template renderings are generated by claude-opus-4.6, which is also one of the nine evaluator models, creating a potential generator/reader confound. The authors address this by checking whether Opus-4.6 systematically outperforms other readers on those conditions (it does not appear to), but the confound cannot be fully eliminated.
The study also focuses entirely on commercial models from three providers. How these results generalise to open-weight models, smaller models, or fine-tuned memory systems is an open question. The 500-question LongMemEval oracle tier is a reasonable evaluation set, but it is a single benchmark with a specific question distribution. The HotpotQA transfer is encouraging, though HotpotQA is a multi-hop retrieval task rather than a conversational memory task, so the transfer is suggestive rather than definitive.
Implications for Evaluation Practice
The practical upshot is clear. Memory and RAG evaluations that do not report the reader-facing artifact are measuring something underspecified. A system that achieves 60% accuracy might be genuinely better at memory retrieval, or it might simply be handing the reader a cleaner artifact. Without controlling for rendering, the two cannot be separated.
RENDER's proposal is actionable: benchmark authors can add a fixed evidence-surface condition as a control; system builders can use the packet ladder to distinguish missing content from unresolved conflict or refusal-triggering surfaces; deployment teams can keep structured internal storage while rendering compact natural language to the reader. The abstention finding alone is worth taking seriously, since three major commercial models refuse to answer from formally structured packets while answering the same facts from natural-language entries. Any deployment that uses formal ledger-style memory representations without testing for this behaviour is carrying unknown risk.
The paper does not argue that structure is always worse than natural language, or that raw dialogue is always best. The budget-matched results show the opposite: when context is constrained, structured resolved packets are dramatically better than recency-truncated raw dialogue. The argument is that rendering is an experimental variable, and uncontrolled experimental variables produce uninterpretable results. That is a methodological point the field should take seriously.
Full paper available at arxiv.org/abs/2608.23568.