Most published comparisons of GraphRAG against vector RAG reach conclusions that are difficult to generalise: the corpus is fixed, the embedder is fixed, and the judge is a single LLM whose consistency is never examined. A new preprint on arXiv, Universal Pathologies, Conditional Consequences: A Triple-Robustness Analysis of RAG for Multi-Hop Traceability, addresses this directly by holding the retrieval architecture constant and systematically varying three orthogonal axes: the embedding model, the corpus, and the faithfulness judge. The result is one of the more carefully controlled RAG evaluation studies to appear in the requirements engineering space, and its findings have implications well beyond aerospace certification.
The motivating domain is DO-178C avionics software certification, where requirements must trace across typed link graphs spanning multiple hops. This is a genuinely hard retrieval task: a query may require chaining derives_from, satisfies, and traces_to edges across two or three document nodes, and citation errors carry regulatory consequence. The authors position this against the broader multi-hop QA literature by also running a cross-corpus replication on MuSiQue Wikipedia paragraph chains, which allows them to separate corpus-specific effects from architectural ones.
Methodology and Experimental Design
The five pipelines under evaluation are vanilla vector RAG, an agentic LangGraph loop, an agentic variant augmented with graph lookup, Microsoft GraphRAG, and a learned adaptive router that selects among the base pipelines per query. All five share the same Chroma vector store and Neo4j Aura graph store, which is an important control: differences in performance cannot be attributed to indexing infrastructure.
The three robustness axes are:
- Embedder: local e5-small (384d) versus Azure text-embedding-3-small (1536d)
- Corpus: DO-178C typed-edge requirements versus MuSiQue Wikipedia paragraph chains
- Judge: paired GPT-5.4 and GPT-4.1 on DO-178C; single GPT-5.4 on MuSiQue
The main matrix runs to 4,440 executions (5 pipelines, 296 hop-stratified queries, 3 seeds, 2 embedders), with an additional 600 cross-corpus runs and 1,200 paired faithfulness judgments. Statistical claims require three conditions to hold simultaneously: BCa bootstrap confidence intervals excluding zero, Holm-corrected Wilcoxon p < 0.05, and Cliff's delta of at least 0.147. This is a stricter joint criterion than most RAG papers apply, and it meaningfully reduces the risk of reporting noise as findings.
Key Findings
The paper makes four numbered contributions, and they are worth unpacking separately because they point in different directions.
C2a: Over-citation is architecturally universal. GraphRAG consistently emits 11 to 15 citation IDs per answer, yielding citation precision of 0.12 to 0.23 across all three experimental settings. Retrieval recall is reasonable at 0.68 to 0.87, so the pipeline is finding relevant material; it is simply attributing answers to far too many sources. This pattern holds regardless of embedder or corpus, which is the important point. It is a property of the GraphRAG architecture, not an artefact of a particular dataset.
C2b: The faithfulness consequence is corpus-conditional. On DO-178C, GraphRAG faithfulness collapses from 74% at 1-hop to 40% at 3-hop. On MuSiQue, the same pipeline rises from 42% to 58% across hops. The mechanism the authors propose is that Wikipedia paragraph chains are topically coherent enough that over-cited paragraphs remain loosely supporting, whereas typed-edge requirements graphs are semantically sparse and structurally precise. An over-cited node in a requirements graph is likely to be genuinely irrelevant; an over-cited Wikipedia paragraph about a related topic may still ground a plausible answer. This distinction between architectural mechanism and corpus-conditional consequence is the conceptual core of the paper.
C1: Stratum-conditional winners are corpus-conditional but embedder-robust. Vanilla RAG wins on 2-hop DO-178C queries; GraphRAG wins on 2-hop MuSiQue queries. This ordering is stable across both embedders, which is a useful finding for practitioners: swapping embedding models does not change which pipeline to deploy, but deploying a pipeline validated on Wikipedia to a requirements traceability task almost certainly will produce the wrong choice.
C3: Single-judge faithfulness is embedder-fragile. GPT-5.4 self-kappa across embedders on the matched 300-tuple subset is 0.137, with verdict changes on 41% of items. This is a striking result. The same judge, evaluating the same pipeline on the same queries, disagrees with itself at near-chance levels when only the embedding model changes. The authors frame this as a kappa paradox effect: when one class dominates (most answers are either faithful or not), raw agreement is high but kappa collapses. The practical implication is that single-LLM faithfulness evaluation is not a reliable signal for comparing retrieval configurations, and paired judge protocols provide only partial mitigation.
C4 (supporting): A logistic regression router on dense embeddings alone reaches macro-F1 0.86 on hop classification. This is presented as a diagnostic instrument rather than a deployable system, but it suggests that hop complexity is largely recoverable from embedding geometry without any explicit structural annotation.
Limitations and Open Questions
The cross-corpus replication on MuSiQue runs only three of the five pipelines, which means the adaptive and standalone agentic results on Wikipedia chains are absent. The authors justify this on grounds that C2a requires only the three-pipeline trio, but it leaves the generalisation of C1 partially incomplete. The MuSiQue subgraph is also relatively small at 3,996 paragraph chunks, and the hop stratification is mapped from MuSiQue's existing structure rather than independently constructed, which introduces some risk of alignment artefacts.
The judge fragility finding (C3) raises a question the paper does not fully resolve: if GPT-5.4 disagrees with itself at kappa 0.137 across embedder conditions, what is the appropriate ground truth for faithfulness in multi-hop requirements traceability? The paper uses paired judges as a partial answer, but the underlying calibration problem remains open. A reference implementation using human expert annotation on a small held-out set would strengthen the evaluation framework considerably.
The DO-178C corpus is described as typed-edge requirements but the paper does not detail how the graph was constructed or whether it reflects a real certification artefact or a synthetic approximation. This matters for practitioners trying to replicate the setup in an actual aerospace programme.
Implications for RAG System Design
The central argument, that triple-robustness across embedder, corpus, and judge is the minimum defensible bar for architecture claims, is well-supported by the data. The field has a habit of reporting GraphRAG comparisons on a single corpus with a single LLM judge, and this paper demonstrates concretely why that produces unstable conclusions. The mechanism-versus-consequence distinction for over-citation is particularly useful: knowing that GraphRAG over-cites everywhere but that this only harms faithfulness in semantically sparse domains gives system designers a principled basis for corpus-conditional pipeline selection.
For anyone building RAG systems over structured technical corpora, the practical takeaway is straightforward. GraphRAG's graph traversal provides genuine recall benefits, but its citation behaviour will degrade precision in any domain where node semantics are precise and non-overlapping. Vanilla or agentic-graph pipelines with explicit hop routing are likely to outperform it on 2-hop structured queries in that regime, and the hop router result suggests this routing decision can be made cheaply from embeddings alone.
The full paper is available at arXiv:2608.05153.