⚠️ LLM Source Citation Verification Simulator

LLM Citation Generation — Plausible References Attached to Clinical Claims

Fluent citations that look real, formatted correctly, generated alongside claims.

• 8–24: Citations per response (typical clinical-answer output)
• >95%: Format plausibility (looks like a real reference)
• <10%: Reviewer spot-check rate (without a dedicated tool)
• Title, journal, year, DOI: Fields per citation (all independently checkable)

Why citations get generated as fluent text

Citations are text tokens, not database lookups — fluency isn't truth.

Existence Verification — Does This Source Even Exist?

First gate: search title, journal, and DOI against real bibliographic records.

• DOI + title lookup: Existence check method (CrossRef, PubMed, journal index)
• 10–20%: Fabrication rate (typical) (depends on model and topic)
• Well-formed, unresolvable: False DOI patterns (looks valid, points nowhere)
• Seconds per citation: Check cost (automatable at scale)

A fabricated citation is a distinct failure mode

No paper, no journal issue, no DOI record — the source is invented.

Relevance Verification — Does the Real Source Support the Claim?

Second gate: read the real source and compare it to the attributed claim.

• 5–15%: Misattribution rate (typical) (of existing citations)
• Right topic, wrong finding: Common pattern (plausible but unsupported)
• Claim-to-abstract comparison: Check method (human or LLM-assisted read)
• Higher than existence check: Detection difficulty (requires reading the source)

Existing sources can still be the wrong evidence

Real paper, real journal — but it never claims what's attributed to it.

Classification — Fabricated, Misattributed, or Verified

Every citation lands in exactly one of three buckets after both checks.

• 3: Outcome classes (fabricated, misattributed, verified)
• Red: Fabricated stamp (source does not exist)
• Orange: Misattributed stamp (real source, wrong claim)
• Green: Verified stamp (real and accurate)

Two independent gates, one final label

Existence and relevance both pass before a citation earns "verified."

Outcome — A Distinct Failure Mode Beyond Fact-Checking

A fabricated citation can slip past reviewers who trust the reference list.

• Yes: Distinct from fact-checking (claim can be true, source fake)
• High: Reviewer trust risk (formatting looks authoritative)
• Independent source verification: Mitigation (every citation, every time)
• Caught failures ≠ 0: Net effect (checks find what reading misses)

Verification is a separate step from accuracy review

Checking the claim is true is not the same as checking the source is real.

A citation can be fabricated even when the underlying medical claim happens to be true — which is exactly why source verification has to run as its own independent check.