LLM-generated biomedical claims are cross-checked against the knowledge graph to catch hallucinations before they reach clinicians
When a clinician poses a question — a drug interaction, a gene-disease association, a dosing threshold — a large language model responds with fluent, confident prose. That fluency is precisely the danger: transformer decoders optimize for plausible next-token continuation, not for factual verification against any external source of truth, so an unsupported claim reads identically to a well-supported one.
Large language models generate text by sampling the next token from a probability distribution conditioned on everything generated so far. This objective — maximize likelihood of plausible continuations — has no built-in mechanism for verifying that a generated statement corresponds to any real-world fact. The model has memorized statistical regularities from its training corpus, not a queryable, updatable store of ground truth.
In biomedicine this is especially hazardous because the space of plausible-sounding but false claims is enormous: drug names, gene symbols, and disease terms combine combinatorially, and the model has strong priors about what a "drug interaction sentence" or a "gene-disease sentence" looks like syntactically, independent of whether the specific relationship is real. Confabulation is most common for long-tail entities (rare diseases, newly approved drugs, orphan gene targets) that were sparsely represented during pretraining.
Temperature and sampling strategy modulate but do not eliminate the problem: greedy decoding reduces variance but can still lock onto a confidently wrong claim early in generation, after which autoregressive conditioning makes the model "double down" to stay consistent with its own prior tokens.
A 2023 JAMA Internal Medicine audit of LLM responses to clinical vignettes found that even top-performing models produced at least one unsupported or fabricated claim in roughly 1 of every 5 multi-sentence answers — despite near-fluent, guideline-consistent phrasing throughout.
Retrieval-augmented generation (RAG) attempts to reduce hallucination by conditioning generation on retrieved documents at generation time. It helps, but does not solve the problem: the model can still ignore retrieved context, misattribute facts between retrieved passages, or blend retrieved and memorized content into a single ungrounded sentence — a failure mode sometimes called "context-unfaithful generation."
Knowledge-graph grounding takes a complementary, post-hoc approach: let the model generate freely (preserving fluency and the ability to synthesize), then independently verify every discrete factual assertion against a structured, curated knowledge graph after the fact. This decouples the generation objective from the verification objective, so verification quality does not depend on whether the model chose to use its context window correctly.
Production clinical NLP pipelines increasingly combine both: RAG to improve first-pass answer quality, and graph-grounded fact-checking as a hard downstream gate before any claim reaches a clinician-facing surface.
Not every sentence needs verification — hedges, questions, and purely stylistic connective tissue carry no checkable content. The claim generation stage of the pipeline filters output into checkable declarative assertions: statements of the form "X causes/treats/interacts-with/is-a Y" that in principle map to an edge or short path in a biomedical knowledge graph.
Heuristics used at this stage include dependency-parse filtering for declarative main clauses, named-entity recognition to confirm at least one biomedical entity is present, and confidence-based triage — claims the model itself expresses with low internal certainty (measured via token-level log-probabilities or self-consistency across sampled generations) are prioritized for verification, since they correlate with higher hallucination risk.
A free-text claim like "metformin reduces HbA1c and is contraindicated in severe renal impairment" bundles multiple independently-checkable facts. Before anything can be verified against a graph, it must be decomposed into atomic triples — the same subject-predicate-object structure that knowledge graphs themselves store as edges — so each assertion can be checked or flagged on its own merits.
Triple extraction — sometimes called Open Information Extraction (Open IE) when unconstrained, or relation extraction when mapped to a fixed schema — converts a sentence into (subject, predicate, object) tuples. In grounding pipelines the predicate vocabulary is typically constrained to the relation types the target knowledge graph actually supports (e.g., treats, causes, contraindicated_in, interacts_with, is_subtype_of), because a triple with an unrecognized predicate cannot be looked up regardless of whether the underlying fact is true.
Modern pipelines use either a fine-tuned sequence-to-sequence extractor (trained on biomedical relation-extraction corpora such as BioRED or ChemProt) or the LLM itself, prompted a second time in a structured "self-extraction" pass to re-express its own prior answer as triples. Self-extraction is attractive because it reuses the same model, but introduces a risk: the model can mis-transcribe its own claim during decomposition, so extraction fidelity is itself validated against the source sentence via round-trip consistency checks.
Round-trip validation — regenerating a natural-language sentence from the extracted triple and checking semantic similarity against the original claim — catches roughly 6-9% of decomposition errors that would otherwise silently corrupt the verification stage.
Raw extracted subjects and objects are surface strings ("Type 2 diabetes," "T2DM," "adult-onset diabetes") that must be normalized to canonical graph node identifiers before any lookup is possible. This entity linking step typically maps free text to standard biomedical vocabularies — UMLS CUIs, MeSH terms, RxNorm codes for drugs, HGNC symbols for genes — using a combination of string similarity, learned entity embeddings, and context disambiguation (the same string can refer to different entities depending on surrounding text).
Ambiguous or unresolvable entities are a primary source of false negatives in the verification stage: if "MS" cannot be confidently resolved to "multiple sclerosis" versus "mitral stenosis," the downstream graph lookup has no node to query, and a true claim can be wrongly flagged simply because linking failed rather than because the fact is false. Production pipelines therefore report a distinct "unresolved entity" state, separate from "unsupported," so downstream review teams can distinguish extraction failures from genuine hallucinations.
Clinical language routinely compounds several facts and negations in one sentence: "Drug A is not recommended with Drug B due to increased bleeding risk, unlike Drug C." Decomposition must correctly split conjunctions, resolve negation scope, and preserve the polarity of each extracted triple — a negated triple queried against the graph as though it were positive would produce a completely inverted, and clinically dangerous, verification outcome.
Current pipelines handle this with negation-scope detection models (e.g., NegEx-style rule systems layered with learned classifiers) applied before triple emission, tagging each triple with a polarity flag that is carried through to the graph lookup stage so that "interacts_with(A,B) = false" and "interacts_with(A,B) = true" are verified against correspondingly different graph evidence.
With claims reduced to normalized triples, each is checked against a biomedical knowledge graph assembled from curated sources — drug-interaction databases, ontologies, and structured literature extractions. Verification asks a precise, answerable question for every triple: does a supporting edge, or a short evidentiary path, exist in the graph — and if so, how strong is that evidence?
The simplest verification case is a direct edge match: the triple (metformin, treats, type_2_diabetes) exists verbatim as a curated edge in the graph, sourced from a structured database like DrugBank or a clinical guideline extraction. This resolves quickly and with high confidence.
Many true claims, however, are not single edges but require multi-hop reasoning: "Drug X may worsen condition Y" might only be verifiable via a path Drug X → inhibits → Enzyme Z → metabolizes → Drug W → contraindicated_in → Condition Y. Path-based verification searches for supporting chains up to a bounded hop count (typically 2-3 hops in production systems, since path space grows combinatorially and semantic drift accumulates with each additional hop), and assigns lower confidence to longer paths, since each additional edge introduces its own uncertainty and every hop compounds the chance that a "supported" path is not really evidence for the specific claim rather than a spurious coincidental connection.
Confidence typically decays geometrically with path length in production systems (roughly a 15-25% confidence discount per additional hop), reflecting empirical calibration studies showing that 3-hop "supporting" paths correspond to true claims only about 60% as often as direct single-edge matches.
A knowledge graph is only as trustworthy as its sources and as current as its last update. Clinical knowledge graphs typically blend several provenance tiers: (1) structured, regularly-updated databases (DrugBank, RxNorm, SIDER for side effects) refreshed on release cycles measured in weeks to months; (2) manually curated ontological relationships (UMLS, SNOMED CT, Gene Ontology) that change slowly and carry high trust; and (3) literature-derived edges extracted automatically from PubMed abstracts via NLP pipelines, which are more current but carry materially higher noise and lower per-edge confidence.
Verification systems weight evidence by provenance tier, not just by presence or absence of an edge — an assertion supported only by a single automatically-extracted literature edge is treated very differently from one confirmed by a structured, manually curated database entry. Graph staleness is a genuine failure mode: a claim about a drug approved or a guideline updated after the graph's last ingestion date will correctly fail lookup even though it may now be true, which is why production systems log a distinct "graph coverage gap" outcome alongside "unsupported."
The strongest verification signal is not merely a missing edge — it is an explicit contradicting edge. If the graph contains (Drug A, safe_with, Drug B) as a curated, high-confidence relationship, and the LLM asserted the opposite, this is flagged with maximum severity and routed for immediate exclusion, distinct from claims that simply have no graph coverage either way.
This three-way outcome space — supported, unsupported (no coverage), contradicted — is essential to communicate accurately downstream: an "unsupported" claim might still be true but outside the graph's scope, whereas a "contradicted" claim is actively dangerous and should never reach a clinician regardless of how fluently it was phrased.
| Product | Indication | Trial Design | Key Result |
|---|---|---|---|
| Pure RAG (retrieval-conditioned generation) | Reduces hallucination during generation | Retrieved passages inserted into prompt context before decoding | Improves fluency-grounded consistency, ~20-30% relative error reduction |
| Post-hoc KG triple verification | Catches hallucinations after generation | Decompose to triples, query graph edges/paths independently of generation | Decision is model-agnostic and auditable per claim |
| Self-consistency / sampling ensembles | Flags internally uncertain claims | Sample the same prompt N times, compare claim agreement across samples | No external KG needed, but weaker precision on confident hallucinations |
| Hybrid RAG + KG gate (this pipeline) | Both reduces and catches hallucinations | RAG improves first-pass quality; graph verification is a hard downstream gate | Highest reported precision/recall balance in clinical NLP benchmarks |
Verification outcomes must be converted into an actionable decision: pass the claim through, flag it for human review, or suppress it outright. Flagging thresholds are not one-size-fits-all — they are tuned against the clinical cost of a false negative (an unsupported claim reaching a clinician) versus a false positive (a true claim wrongly suppressed for lack of graph coverage).
Flagging is fundamentally a binary classification problem layered on top of graph verification confidence scores, and it inherits all the calibration challenges of clinical decision thresholds. Setting the strictness threshold too loose lets ungrounded claims slip through to the clinician (a false negative on hallucination detection — the dangerous failure mode). Setting it too strict suppresses true claims that simply fall outside current graph coverage (a false positive that erodes clinician trust in the system and can withhold useful information).
Production systems typically report threshold performance as an ROC or precision-recall curve swept across the verification confidence score, and select an operating point that favors recall on the "contradicted" class heavily (near-zero tolerance for confidently wrong claims reaching output) while accepting a higher false-flag rate on the "unsupported, no coverage" class, since a spuriously flagged true claim is recoverable via human review while an unflagged hallucination is not.
A 2024 multi-site clinical NLP evaluation found that tuning the flagging threshold to prioritize hallucination recall over precision reduced dangerous unflagged errors by 82%, at the cost of roughly doubling the human-review queue — a tradeoff most deploying institutions accepted given the asymmetric clinical stakes.
Rather than a binary pass/fail, mature pipelines assign a severity tier to each flagged triple: contradicted (graph evidence directly opposes the claim — auto-suppress, highest severity), unsupported (no graph path found within the hop budget — flag for review), low-confidence-path (only a long, weakly-scored multi-hop path found — flag with lower urgency), and coverage-gap (entity or relation type not represented in the graph at all — flag as "unverifiable" rather than "false").
This tiering lets downstream systems and human reviewers triage efficiently: contradicted claims are pulled automatically before any human sees the output, while coverage-gap claims might be routed to a secondary verification pathway (e.g., a citation-search fallback against primary literature) rather than blocked outright.
Verification strictness is the operational knob institutions tune per use case. A low-strictness setting accepts single weakly-scored paths as sufficient evidence, maximizing claim throughput but allowing more marginal claims through ungrounded. A high-strictness setting requires direct, high-provenance edge matches and rejects anything relying on long inferred paths, minimizing hallucination leakage at the cost of flagging more true-but-uncovered claims.
In practice, high-stakes contexts (dosing, contraindications, drug interactions) are configured with maximum strictness regardless of overall system setting — these claim categories use a fixed, non-adjustable floor — while lower-stakes informational claims (general mechanism-of-action background) can tolerate a looser threshold to preserve response completeness.
The final stage reassembles only the claims that survived verification into a coherent answer, each one carrying a traceable citation back to its supporting knowledge-graph evidence. This closes the loop: the clinician receives an answer that is not merely fluent, but auditable — every assertion can be traced to a specific, inspectable source.
Once flagged triples are removed, the remaining verified triples must be reassembled into readable prose — not merely listed as raw graph edges, since a bulleted list of subject-predicate-object tuples is unusable in a clinical workflow. A generation pass (often the same LLM, now constrained to only the verified triple set as its permissible content) re-expresses the surviving facts in natural language, with each sentence carrying an inline citation to the specific graph edge or path that supports it — analogous to a reference-linked literature citation, but pointing to a structured knowledge-graph provenance record instead of a paper.
This constrained regeneration step is itself validated: the same round-trip consistency check used during decomposition confirms that the regenerated sentence does not silently reintroduce unsupported content that was not present in the verified triple set, closing a subtle failure loop where generation could otherwise "helpfully" restore removed information from its own memorized priors.
Because the second-pass generation is constrained to only verified triples as permissible source content, measured hallucination rate in the final delivered answer drops from an unconstrained baseline of 8-39% to under 2% in benchmark evaluations of the full pipeline.
A grounded answer is necessarily incomplete relative to the model's original unconstrained response — flagged claims are removed, not silently replaced. Well-designed systems surface this explicitly rather than hiding it: a visible note such as "2 additional claims could not be verified against current knowledge sources and have been withheld" preserves clinician trust and gives an honest signal about coverage limits, rather than presenting a shortened answer as if it were complete.
Some deployments offer an expandable "unverified claims" panel showing suppressed content with its flag severity, allowing a clinician with independent domain expertise to judge withheld claims on their own terms rather than losing that information entirely — treating the system as a triage assistant rather than a silent censor.
Deployed grounding pipelines are evaluated continuously, not just at launch. Clinician feedback on flagged and delivered claims (agree / disagree / uncertain) feeds back into two places: threshold recalibration (adjusting the strictness operating point as real-world precision/recall data accumulates) and knowledge graph maintenance (systematic disagreement on a specific claim category often reveals a graph coverage gap or a stale edge that needs curator attention).
This creates a virtuous cycle distinct from static fact-checking: the graph itself improves as an artifact of the verification pipeline being used in production, and flagging thresholds adapt to observed real-world error patterns rather than remaining fixed at their initial calibration.
Graph-grounded verification substantially reduces but does not eliminate risk. It cannot catch errors in claims the graph itself gets wrong (curation errors, stale edges), cannot verify claims about genuinely novel findings not yet represented in any structured source, and cannot assess claim quality beyond factual support — a verified-true claim can still be poorly prioritized, out of clinical context, or misleading by omission even when every individual assertion is technically grounded.
For this reason, graph-grounded LLM output in clinical settings is near-universally deployed as a decision-support tool requiring clinician review, not as an autonomous decision-maker — the grounding pipeline raises the floor on factual reliability substantially, but does not replace clinical judgment.