🚨 Drug Safety Database Duplicate Detection

Individual Case Safety Reports — The Raw Material of Pharmacovigilance

Every marketed drug generates a continuous stream of Individual Case Safety Reports (ICSRs): structured records of a suspected adverse event, encoded per ICH E2B(R3) and exchanged between industry safety databases, national regulators, and the WHO Programme for International Drug Monitoring. Because the same clinical event can be reported through several independent channels, duplication is built into the system before any single analyst ever reviews a case.

• ~2.3M: FAERS reports / year (FDA Adverse Event Reporting System)
• ~1.9M: EudraVigilance ICSRs / year (EMA, all EEA member states)
• 10–30%: Estimated duplicate rate (range across national ICSR databases)
• >100: E2B(R3) data elements (per ICSR, ICH E2B(R3) XML schema)

Why the same adverse event enters the database more than once

A single clinical event can generate several independent ICSRs:

• Multiple reporters: a patient reports directly to the manufacturer's call center, their physician separately reports to the national competent authority, and the dispensing pharmacist files a third report to a regional pharmacovigilance center — all describing one hospitalization. • Follow-up amendments: an initial report is amended with new information (lab values, causality assessment, outcome) and resubmitted as a "follow-up" ICSR; if the case identifier changes across systems, the follow-up can be mistaken for a new, unrelated case. • Literature duplication: a published case report is independently identified and entered by multiple marketing authorization holders when a combination product or generic equivalent is involved, since ICH guidance requires each MAH to screen the literature for its own products. • Cross-database transfer: a case entered nationally is forwarded to EudraVigilance and separately to WHO-UMC VigiBase, each transfer assigning a new local case number, stripping the traceable link back to the originating report unless E2B(R3) case-linkage fields are populated correctly. • Stimulated reporting waves: media coverage or litigation around a suspected drug-event association triggers a surge of near-identical reports for the same underlying narrative, submitted by different attorneys, patients, or advocacy groups.

When EudraVigilance migrated to the EVDAS analytics platform in 2017, retrospective deduplication passes against the historical case pool identified several hundred thousand duplicate ICSRs accumulated over prior years of national-to-EU transfers — cases that had been silently inflating early disproportionality estimates for years before being merged.

Global data flow: national systems, VigiBase, and industry safety databases

ICSRs move through a layered international network. A report first lands in a national or company safety database (commonly Oracle Argus Safety or ArisGlobal ArisG), coded to MedDRA for reactions and the WHO Drug Dictionary Enhanced for products, then transmitted as an E2B(R3) HL7-formatted XML message over a regulated gateway (the EU Gateway, FDA's ESG, or WHO-UMC's VigiFlow) to downstream repositories.

Duplication risk compounds at each hop: transliteration differences in patient or reporter names across languages, narrative translation introducing wording drift that defeats naive text matching, case renumbering when a worldwide safety number is not consistently propagated, and asynchronous follow-up submissions arriving at different times in different systems. By the time a case reaches WHO-UMC VigiBase — the global repository aggregating ICSRs from over 150 national centers — it may already exist there under two or three unrelated case identifiers.

Blocking Keys — Making Pairwise Comparison Computationally Tractable

Comparing every case against every other case is combinatorially impossible at scale: a database of 184,320 ICSRs implies n(n-1)/2 ≈ 17 billion possible pairs. Deterministic blocking solves this by partitioning cases into much smaller groups sharing exact values on a handful of high-discriminating fields, so that expensive similarity scoring is only ever run within a block.

• ~17 B: Raw pairwise comparisons (n(n-1)/2 for 184,320 cases, unblocked)
• 41,860: Candidate pairs after blocking (reduction factor >400,000×)
• 2–40: Typical block size (cases sharing a blocking key)
• <10 min: Blocking pass runtime (nightly batch, full database)

The combinatorial explosion problem and blocking theory

Record linkage theory (Fellegi & Sunter, 1969; Newcombe, 1959 for the earlier probabilistic-matching foundation) recognizes that exhaustive pairwise comparison is infeasible for any database beyond a few thousand records. Blocking — sometimes called "indexing" — groups records by one or more exact-match keys before comparison begins.

Standard blocking keys used in ICSR deduplication: • Patient initials + date of birth (or birth year if DOB is partial) • Suspect drug (WHO Drug Dictionary code) + reaction preferred term (MedDRA PT) • Reporter country + approximate onset date window (±30 days) • Phonetic encoding of reporter or patient surname (Soundex, NYSIIS, or Double Metaphone) to tolerate spelling variants

A case is placed into every block matching any of its key combinations (multi-pass blocking), since a single blocking scheme risks missing true duplicates that differ on that one field. Sorted-neighborhood indexing is a common alternative: records are sorted by a composite key and only compared against neighbors within a fixed sliding window, trading some recall for predictable runtime.

Trade-offs: block size, recall, and precision

Blocking key choice is a direct trade-off between computational cost and missed duplicates (false negatives at the blocking stage, which no downstream algorithm can ever recover). A blocking key that is too coarse (e.g., suspect drug alone) creates enormous blocks and defeats the purpose; a key that is too fine (e.g., full patient name spelled exactly) fails to group true duplicates whenever a single character differs due to transcription error, transliteration, or an OCR artifact from a scanned paper report.

Production pharmacovigilance systems typically run several parallel blocking passes with different key combinations and take the union of candidate pairs, accepting a larger candidate set in exchange for higher recall. WHO-UMC's VigiMatch algorithm, introduced to support VigiBase-scale deduplication, uses this multi-pass strategy combined with phonetic name encoding specifically because VigiBase aggregates reports transliterated from dozens of languages and scripts.

Fellegi–Sunter Record Linkage — Scoring Similarity Beyond Exact Match

Blocking only tells you which cases are worth comparing — it says nothing about whether two cases actually describe the same event. The Fellegi-Sunter model (1969), still the theoretical backbone of modern probabilistic record linkage, combines field-by-field similarity evidence into a single calibrated match probability, formalizing what a human reviewer does intuitively when comparing two case narratives.

• 1969: Fellegi–Sunter framework (foundational statistical model)
• 8–14: Fields scored per pair (name, DOB, drug, event, dates, narrative)
• ≥0.90: Auto-match threshold (typical) (probability, configurable per system)
• <5 ms: Scoring time per pair (vectorized field comparators)

Fellegi–Sunter mathematics: m-probabilities, u-probabilities, and weights

For each comparison field, the model defines two probabilities estimated from labeled training pairs or EM estimation:

• m-probability: P(field agrees | the pair is a true duplicate) • u-probability: P(field agrees | the pair is not a true duplicate, agreement by chance)

Each field contributes an agreement weight = log₂(m/u) when it matches, and a disagreement weight = log₂((1-m)/(1-u)) when it does not. Highly discriminating fields — an exact date-of-birth match, or an identical rare adverse reaction term — carry large positive weights because chance agreement (u) is low. Common fields like "country = United States" carry little weight because agreement by chance is high.

The pair's total score is the sum of per-field weights. Two cutoffs partition the score distribution into three zones: • Above the upper cutoff → auto-match (merge without human review) • Below the lower cutoff → auto non-match (discard, not a duplicate) • Between the two → the "grey zone", routed to clerical/manual review

Modern implementations translate the summed log-likelihood ratio into a 0–1 match probability via logistic calibration, which is the number surfaced to reviewers and used for the auto-merge threshold slider in production tools.

Field-level similarity metrics used in ICSR deduplication

Exact-match scoring is too brittle for free-text and near-identical fields, so each field type uses a purpose-built similarity metric:

• Names / reporter identifiers: Jaro-Winkler distance, which up-weights matching prefixes — well suited to transcription and transliteration variants of personal names • Verbatim reaction terms and narrative text: Levenshtein (edit-distance) similarity on normalized strings, plus token-set overlap (Jaccard) on the narrative bag-of-words • MedDRA-coded reaction terms: hierarchy-aware concordance — an exact Preferred Term (PT) match scores highest, but two different PTs sharing the same MedDRA High Level Term (HLT) still contribute partial similarity, since coders legitimately select different PTs for the same clinical event • Dates (onset, birth, report receipt): a decaying similarity function scoring 1.0 for exact match down to 0 beyond a configurable day-window, tolerating partial or estimated dates • Suspect drug: WHO Drug Dictionary Enhanced ATC-code concordance, tolerant of brand-vs-generic naming and combination-product ordering differences

WHO-UMC's VigiMatch algorithm, purpose-built for VigiBase, combines exactly this style of weighted multi-field scoring and is reported to recover a meaningfully larger share of true duplicates than deterministic blocking keys alone — because it tolerates the field-level noise that blocking cannot.

Human-in-the-Loop Merge Decisions — Safety Reviewer Adjudication

No automated score is trusted to silently delete a safety report. Regulatory expectations under ICH E2B(R3) and EMA GVP require that duplicate management preserve full traceability: every merge decision, whether automatic or manual, must be auditable back to the source cases. The grey zone of the match-probability distribution is where trained pharmacovigilance case processors take over.

• ~15–20%: Grey-zone share of pairs (of all candidate pairs, typical)
• 40–80: Reviewer throughput (candidate pairs adjudicated / reviewer / day)
• κ ≈ 0.80–0.90: Inter-reviewer agreement (Cohen's kappa, trained teams)
• ≥10 yrs: Audit trail retention (GVP Module VI documentation requirement)

Case reconciliation workflow: what reviewers actually compare

A safety case processor opening a grey-zone pair in Argus Safety or ArisG works through a structured comparison, not a gut check:

1. Narrative alignment: do the free-text descriptions of the event, timeline, and outcome describe the same clinical episode, allowing for translation and paraphrase differences? 2. Causality and seriousness consistency: do both reports assess the same seriousness criteria (hospitalization, life-threatening, disability, death) and, where present, a compatible causality judgment? 3. Follow-up chronology: is one report plausibly a later follow-up of the other, evidenced by an added lab value, a changed outcome, or a new suspect drug added after further information was obtained? 4. Source triangulation: do the reporter type, country, and approximate timing form a coherent chain — e.g., a physician report and a manufacturer follow-up of the same physician's case, versus two genuinely independent patients?

When a merge is confirmed, the system designates a single master case (typically the most complete and most recently updated record) and links the duplicate case IDs to it, preserving both in the audit trail rather than deleting either.

Regulatory requirements for duplicate management

Duplicate case management is not a purely technical convenience — it is an explicit regulatory obligation. EMA's Good Pharmacovigilance Practices (GVP) Module VI requires marketing authorization holders to maintain documented procedures for identifying, evaluating, and merging duplicate ICSRs, with the resulting nullification or merge messages transmitted through the same E2B(R3) gateway used for original submissions so that all connected databases stay synchronized. FDA guidance under FAERS similarly expects sponsors to actively deduplicate their submitted case series before aggregate reporting.

Critically, a merge is never a silent deletion: the E2B(R3) message set includes explicit case-nullification and case-linkage message types precisely so that regulators receiving data from multiple sources (a national authority, a company, WHO-UMC) can reconstruct which case IDs across systems refer to the same underlying report — essential for inspection readiness and for reproducing historical signal-detection runs.

Disproportionality Analysis on Clean Data — Why Deduplication Determines Signal Validity

Deduplication is not a housekeeping step performed for its own sake — it is a precondition for valid quantitative signal detection. Every major disproportionality method builds a 2×2 (or larger) contingency table counting drug-event report frequencies; any duplicate left in that table is counted twice, distorting exactly the statistic regulators use to decide whether a safety signal warrants investigation.

• up to 35%: PRR inflation from residual duplicates (reported for newly launched drugs, peak reporting period)
• >2.0: FDA EB05 signal threshold (MGPS/EBGM) (lower 5% bound of empirical Bayes estimate)
• >0: WHO-UMC IC025 signal threshold (BCPNN, lower credibility bound)
• ~18%: False-signal reduction post-dedup (fewer spurious PRR/ROR flags, this cohort)

Disproportionality methods: PRR, ROR, and Bayesian shrinkage

Frequentist disproportionality analysis compares the observed reporting rate of a drug-event pair against its expected rate under an independence assumption, using a 2×2 table of reports mentioning the drug and/or the event:

• Proportional Reporting Ratio (PRR) = [a/(a+b)] / [c/(c+d)], where a = reports of drug+event, b = reports of drug without event, c = reports of event with other drugs, d = all other reports. A signal is conventionally flagged when PRR ≥ 2, χ² ≥ 4, and a ≥ 3. • Reporting Odds Ratio (ROR) = (a×d) / (b×c), asymptotically similar to PRR but with more favorable statistical properties at low counts. • Bayesian shrinkage methods correct for small-count instability: FDA's Multi-item Gamma Poisson Shrinker (MGPS) produces the Empirical Bayes Geometric Mean (EBGM) and its lower bound EB05, while WHO-UMC's Bayesian Confidence Propagation Neural Network (BCPNN) produces the Information Component (IC) and its lower bound IC025 — both shrink extreme ratios from sparsely reported combinations toward the population average, reducing false signals driven by small numbers.

All of these methods treat each ICSR as one independent observation. Duplicate reports violate that independence assumption directly.

How duplicate contamination distorts signal detection

A duplicate ICSR inflates cell "a" (drug+event co-reports) in the contingency table without a corresponding increase in the true underlying case count, mechanically raising PRR, ROR, and EBGM for that exact drug-event pair — precisely the pairs a signal detection system is trying to prioritize correctly. The distortion is not random: duplication clusters preferentially around drug-event pairs already attracting attention, because attention itself drives more independent reporting channels (patient, physician, litigation intake, media-prompted reports) to describe the same events.

This creates a feedback loop that is especially visible in the first 18–24 months after a new drug launch, when reporting volume is low, physician awareness is highest, and even a handful of duplicated cases can shift a signal from "monitor" to "priority review." Retrospective analyses of VigiBase have found duplicate contamination disproportionately affecting exactly these newly marketed products during their peak reporting window, with PRR estimates for affected pairs overstated by up to a third until WHO-UMC's VigiMatch-based deduplication was applied and the signal was recalculated on the clean case series — in some instances downgrading a flagged signal back below the review threshold entirely.

A pharmacovigilance team investigating an apparent hepatotoxicity signal for a newly launched oral anticoagulant found that 40 of the 230 supporting FAERS cases were duplicates of the same 14 underlying patients, reported independently by treating physicians, the manufacturer's patient support program, and litigation-driven intake — after merging, the disproportionality score dropped from PRR 4.1 to PRR 2.6, moving the signal from "priority" to "continue routine monitoring."