One of the more persistent problems in applying large language models to biomedical research is knowing when to trust them. ChatGPT can generate plausible-sounding associations between diseases, genes, proteins, and drugs at scale, but plausible is not the same as correct. Hallucination in this domain is not just a technical inconvenience; it carries real consequences for downstream hypothesis generation and knowledge curation. This paper, available at arXiv:2605.30400, addresses this directly by presenting a structured protocol for generating, validating, and verifying disease-centric biomedical associations from ChatGPT, with a cross-model majority voting mechanism and a RAG-based semantic verification layer sitting on top.
The work is published in STAR Protocols, which is significant context. STAR Protocols publishes detailed, reproducible methodological papers rather than results-driven studies. The contribution here is the workflow itself, not a particular finding about any one disease. That framing matters when reading the paper critically.
Key Contributions
The protocol makes several distinct methodological contributions worth unpacking:
- Structured association generation: ChatGPT is prompted to generate disease-centric associations spanning multiple biological entity types, including genes, drugs, pathways, and phenotypes. The protocol specifies how prompts are structured to elicit consistent, parseable outputs.
- Ontology-based entity validation: Generated biological entities are checked against established biomedical ontologies. This catches obvious errors where ChatGPT invents or garbles entity names, but exact-match validation has well-known limitations when synonyms, abbreviations, or alternative nomenclatures are involved.
- Self-consistency via majority voting: The same queries are run across multiple ChatGPT model variants, and associations that appear consistently across models are treated as more reliable. This borrows from the self-consistency literature in LLM reasoning, where sampling multiple outputs and aggregating them improves factual accuracy.
- RAG-enabled semantic verification: To address the shortcomings of exact-match ontology checking, the protocol introduces a RAG layer powered by open-source LLMs. This allows the system to retrieve relevant literature and use an LLM to assess whether a generated association is supported by published evidence. Critically, this means one set of LLMs is being used to evaluate the outputs of another, which the authors frame as a mechanism for hallucination exposure.
Methodology in Detail
The pipeline follows a logical sequence. First, ChatGPT generates candidate associations for a target disease. Second, the biological entities within those associations are validated against ontologies such as MeSH, Gene Ontology, or similar resources. Third, the cross-model majority voting step filters associations by consistency. Fourth, the RAG verification step queries a literature corpus, retrieves relevant passages, and passes them to an open-source LLM to determine whether the association is textually supported.
The semantic verification step is the most technically interesting component. Exact-match ontology validation will correctly reject a hallucinated gene name, but it fails when ChatGPT produces a real gene name in a false associative context, or uses a valid synonym that the ontology does not index. RAG-based verification is better suited to catching these cases because it grounds the check in actual literature content rather than vocabulary membership alone.
The use of open-source LLMs for verification rather than ChatGPT itself is a deliberate design choice. Using the same model to verify its own outputs risks circular confirmation of systematic biases. Introducing a different model family as the verifier adds a degree of independence, though it does not fully escape the problem since open-source models share training data lineage with proprietary ones to varying degrees.
Limitations and Open Questions
Several methodological tensions are worth examining carefully. The majority voting approach assumes that consistency across ChatGPT model variants is a reasonable proxy for factual accuracy. This is a reasonable heuristic, but ChatGPT model variants within the same family share substantial architectural and training data overlap. A false association that appears in the pretraining corpus consistently will be reproduced consistently across variants, meaning majority voting may reinforce systematic errors rather than catch them. The protocol would benefit from explicit discussion of this failure mode.
The RAG verification layer introduces its own assumptions. The quality of verification depends heavily on the retrieval corpus, the retrieval mechanism, and the open-source LLM's ability to correctly interpret retrieved passages. If the retrieval corpus has coverage gaps for a particular disease area, the verifier will default to uncertainty or, worse, confabulate based on partially relevant passages. The protocol describes the use case at a high level but does not appear to provide detailed benchmarking of the verifier's precision and recall against a curated ground truth.
There is also a question of generalisability. The protocol is presented as disease-agnostic, but the complexity of biomedical association generation varies enormously across disease areas. Rare diseases with sparse literature will stress the RAG verification step in ways that well-studied conditions like type 2 diabetes will not. A single use case demonstration, while illustrative, leaves open how the workflow degrades under these conditions.
From a reproducibility standpoint, the reliance on ChatGPT via API introduces a temporal validity problem. ChatGPT models are updated without versioned guarantees, meaning the same prompts may produce different outputs over time. This is a known issue in LLM-based research protocols and the authors should address it explicitly, ideally by specifying exact model versions and snapshot dates.
Implications for the Field
Despite these limitations, the paper addresses a genuine gap. Most work on LLM hallucination in biomedicine focuses on evaluation after the fact, asking whether a model's output is correct. This protocol is oriented toward building verification into the generation workflow itself, which is a more operationally useful framing for researchers who want to actually use ChatGPT as a knowledge extraction tool rather than simply characterise its failure modes.
The cross-model verification idea, using open-source LLMs to audit proprietary model outputs via RAG, is a practical architecture that others can adapt. It sidesteps the cost and access limitations of running everything through a single proprietary API while maintaining a degree of independence in the verification step. As open-source models continue to close the performance gap with proprietary ones, this architecture becomes more attractive.
For biomedical knowledge graph construction, drug repurposing pipelines, and hypothesis generation tools, having a structured protocol for assessing ChatGPT output quality is directly useful. The work is most valuable as a starting point for groups building these systems, rather than as a validated benchmark in its own right. The STAR Protocols format is appropriate for that purpose.
The full paper and supplementary information are available at https://arxiv.org/abs/2605.30400, with the journal version published in STAR Protocols 2026 via doi:10.1016/j.xpro.2026.104533.