Resolving duplicate and conflicting patient records scattered across sources and time into one unified Master Patient Index
A single patient's clinical footprint is rarely captured by one system. Across a lifetime of care, the same person accumulates distinct, disconnected chart entries in hospital EHRs, independent labs, pharmacies, payer claims systems, and specialty clinics — each with its own identifier scheme and none of them aware the others exist.
Admit-Discharge-Transfer (ADT) messages remain the workhorse of patient identity propagation in US hospitals, despite HL7v2 dating to the late 1980s. An A01 (admit) or A04 (registration) event carries the PID segment — patient identifiers, name, DOB, sex, address — and fires the moment a registrar keys in a new encounter. These feeds are near-real-time, pushed over MLLP or a message broker the instant a chart is opened.
Contrast this with claims and payer data, which typically arrives in nightly or weekly batch files (837/835 X12 transactions, or bulk FHIR exports), and lab or imaging data trickling in from independent reference labs on their own release cadence. A single patient's longitudinal picture is therefore assembled from feeds moving at wildly different speeds, with no guarantee any two of them agree on how to spell the patient's name or which identifier scheme to trust.
Modern FHIR-based exchange (Patient resource, R4) improves structure — identifier arrays with explicit system URIs, standardized name and telecom datatypes — but does not eliminate the underlying problem: each source system still generates its own local identifier, and FHIR's identifier.system field only tells you where an ID came from, not whether it matches an ID from anywhere else.
A large integrated delivery network ingesting from 5–8 source systems commonly discovers that 1 in 10 patients already has two or more unreconciled chart entries before any matching logic runs at all.
Every contributing system was built to solve a local problem, not a network-wide identity problem. A hospital EHR assigns a Medical Record Number (MRN) scoped to that facility; a second hospital in the same health system, even running the same EHR vendor, often maintains an entirely separate MRN sequence. A regional lab assigns its own accession-linked patient ID. A payer identifies members by subscriber ID, which changes when someone switches insurance plans — severing the very key that might have linked their claims history.
This heterogeneity creates two distinct hazards:
• Identifier collision: two different systems independently reuse the same numeric ID for two different patients, purely by coincidence of sequential ID generation, which naive matching can misread as a true link • Identifier drift: the same patient receives a new local ID whenever they re-enter a system after a lapse — a common occurrence at urgent care clinics and emergency departments that do not reliably capture prior visit history
Because no field is guaranteed unique, present, or stable across every source, reconciliation cannot rely on any single identifier. It must instead treat each incoming fragment as an unverified claim about a person's identity, to be corroborated or refuted by everything else known about that person.
At this stage, the reconciliation engine deliberately performs no linkage. Every incoming fragment — an ADT feed record, a FHIR Patient resource, a claims header — is staged into a raw landing zone exactly as received, tagged only with its source system and arrival timestamp. This is a governance decision as much as a technical one: matching logic that runs before data is normalized and validated tends to compound errors rather than resolve them.
Standard ingestion hygiene at this point includes:
• Schema validation against the source's declared standard (HL7v2 segment structure, FHIR resource profile conformance) • Basic normalization — case-folding names, standardizing date formats, stripping punctuation from phone and SSN fields • Preservation of full source provenance: which system, which interface engine, which timestamp, so every downstream match decision remains traceable back to its origin
Only once fragments are validated and provenance-tagged does the pipeline hand them to deterministic matching. Skipping this staging discipline is one of the most common root causes of downstream false merges in production MPI systems.
Deterministic matching links two records only when specific identifying fields agree exactly — or agree after light, rule-based normalization. It is the first line of defense in every Enterprise Master Patient Index (EMPI), prized for speed and near-zero false-positive risk, but it silently misses every patient whose data contains a typo, a legal name change, or a missing identifier.
Deterministic matching runs as an ordered chain of exact-match rules, evaluated from most to least reliable, stopping at the first rule that fires:
• Rule 1: exact SSN match (highest confidence, but SSN is frequently absent, masked to last-4, or intentionally withheld by patients) • Rule 2: exact MRN match within the same source system, or a cross-referenced enterprise ID already assigned by a prior linkage event • Rule 3: exact match on DOB + full legal name + sex, sometimes combined with a partial phone or ZIP match as a tiebreaker • Rule 4: exact match on a secondary government ID (driver's license, Medicare Beneficiary Identifier) when captured
Before any rule runs, records are partitioned into 'blocks' — subsets sharing a coarse key like DOB + first three letters of last name + ZIP — so the engine only compares candidate pairs within the same block rather than performing an O(n²) comparison across the entire population. Blocking strategy is itself a tuning problem: blocks too narrow miss true matches whose block key fields disagree slightly; blocks too wide create unmanageable comparison volume.
Deterministic rules fail in entirely predictable ways, all of which bias toward missed matches rather than incorrect ones — the defining safety property of the approach:
• Typos and transcription errors: 'Kathryn' vs 'Catherine', a transposed digit in DOB entered by a rushed registrar • Legal name changes: marriage, divorce, or gender-affirming name changes break any exact-name rule permanently unless a prior-name field is captured • Missing SSNs: undocumented patients, minors, and privacy-conscious patients frequently decline to provide SSN, silently disqualifying the highest-confidence rule • Nicknames and cultural naming conventions: 'Bob' vs 'Robert', multi-part surnames recorded inconsistently across systems, generational suffixes (Jr./Sr./III) dropped or misplaced
Because deterministic rules only ever link records that agree exactly, they essentially never produce a false merge — the error mode is exclusively false negative (two records for the same patient that remain unlinked). This asymmetry is precisely why deterministic matching runs first: it establishes a trustworthy backbone of confirmed links before any fuzzier, riskier logic is introduced.
Commercial EMPI platforms (identity-resolution vendors serving the EHR interoperability market broadly follow this same deterministic-then-probabilistic architecture) report that deterministic rules alone typically resolve 35–45% of true duplicate pairs, leaving the remainder to probabilistic scoring.
An EMPI sits logically above every source EHR and ancillary system, maintaining an Enterprise Identifier (EID) that cross-references every local MRN known to belong to the same person. When a new record arrives, the EMPI engine runs it through blocking and the deterministic rule chain against the existing identity graph; a confirmed match appends the new local ID as a cross-reference under the existing EID, while a non-match provisions a new EID pending further review.
The commercial identity-resolution space includes platforms built specifically around this cross-reference model, generally differentiated along:
• Deployment model: hosted enterprise EMPI vs. cloud identity-resolution service • Matching engine sophistication: pure deterministic vs. hybrid deterministic/probabilistic scoring • Steward workflow tooling: how flagged possible-matches are queued, visualized, and adjudicated by data quality staff • Interoperability surface: native HL7v2 ADT listener, FHIR Patient $match operation support, bulk reconciliation batch jobs
Regardless of vendor, the deterministic layer's job is narrow and well-defined: link only what can be linked with near-certainty, and pass everything else downstream rather than guessing.
For the fraction of records deterministic rules cannot resolve, probabilistic matching applies a statistical scoring model to weigh partial, fuzzy agreement across many demographic fields simultaneously — trading some risk of false positives for a substantial recovery of true matches that exact rules would otherwise miss entirely.
Ivan Fellegi and Alan Sunter's 1969 paper established the statistical foundation nearly every modern probabilistic matcher still uses. For each comparison field (name, DOB, address, phone), the model computes two conditional probabilities:
• m-probability: the chance the field agrees given the pair truly is the same person (accounting for realistic data entry noise) • u-probability: the chance the field agrees purely by coincidence given the pair is actually two different people
Each field's contribution to the overall match score is its log-likelihood ratio, log(m/u) when the field agrees and a corresponding penalty when it disagrees. Fields that are highly discriminating when they agree but also highly likely to agree by chance (like sex, or common surnames) receive smaller weight than rare, highly specific agreements (an uncommon last name, an exact DOB match). Summing weighted contributions across all compared fields produces a composite score for the candidate pair.
The resulting score is then partitioned into three decision bands using two calibrated cutoffs:
• Score ≥ upper cutoff → automatic match • Score between cutoffs → possible match, routed to a human steward review queue • Score ≤ lower cutoff → automatic non-match
Individual field comparisons themselves use similarity metrics tolerant of realistic data entry variation rather than requiring exact agreement:
• Jaro-Winkler distance: a string-similarity metric (0–1 scale) that rewards matching characters in the same relative order and gives extra weight to agreement in the first few characters — well-suited to names, where typos and truncation are common but prefixes tend to be entered correctly • Soundex / NYSIIS phonetic encoding: reduces a name to a phonetic code so that 'Smith' and 'Smyth' collapse to the same key, catching variant spellings that share pronunciation • DOB proximity scoring: exact DOB match scores highest, but a single-digit transposition (e.g., day/month swapped, a common data entry error) or a near match within a few days can still contribute partial weight rather than zero • Address and ZIP matching: full street-address agreement scores highest; ZIP-only or city-only agreement contributes smaller partial weight, useful for patients who have moved but retained regional ties
Threshold tuning is a direct precision/recall tradeoff: the Match Threshold control in this simulation raises or lowers the score cutoff separating automatic matches from the possible-match review band — pushing it toward Strict reduces false merges but pushes more true matches into manual review or non-match; pushing it toward Loose recovers more true matches automatically but increases false-merge risk.
Lowering the match threshold from a Strict to a Loose setting can roughly double the automatic-match recovery rate — but the same shift typically increases false-merge incidence several-fold, illustrating why threshold governance, not just algorithm choice, is a clinical safety decision.
Every probabilistic matching threshold implicitly encodes a clinical risk tradeoff, and the two failure modes are not symmetric in consequence:
• False negative (missed true match, records stay split): a clinician sees an incomplete history — perhaps missing a documented allergy or a prior adverse drug reaction recorded under the unlinked fragment. Dangerous, but the record itself remains internally consistent and the gap is often visible as 'incomplete history' rather than actively misleading.
• False positive (false merge, two different patients combined): a clinician sees a single chart that blends two people's allergies, medications, diagnoses, and history. This is the more insidious error — the record looks complete and authoritative, giving no visual signal that it is wrong, and can directly drive a wrong-patient medication order or procedure.
Because of this asymmetry, mature MPI programs deliberately bias probabilistic thresholds conservative — accepting a higher rate of possible-match review queue volume in exchange for a lower false-merge rate — and treat every automatic match at the boundary of the decision bands as a candidate for periodic retrospective audit.
Confirming that two record fragments belong to the same patient is only half the problem. Once merged, those fragments frequently disagree — different addresses, different allergy lists, different medication histories captured at different points in time — and the reconciliation engine must decide, field by field, which value survives into the single golden record a clinician will actually see.
When merged fragments disagree on a field's value, a survivorship rule set determines the outcome. No single strategy is correct for every field type, so mature MPI governance applies different rules per field category:
• Most-recent-wins: appropriate for fields that genuinely change over time and should reflect current state — address, phone number, insurance coverage, marital status. The most recently captured value simply replaces older ones. • Most-trusted-source-wins: appropriate when one contributing system is authoritative for a domain — for example, the prescribing EHR's medication list may be trusted over a pharmacy claims feed that only reflects fills, not intent, or a specialty allergy-and-immunology system may be trusted over a general primary-care allergy list. • Union / accumulate, never overwrite: appropriate for fields where losing information is dangerous even if it looks contradictory — allergy lists and problem lists are typically unioned rather than replaced, so a documented penicillin allergy from any source is never silently dropped. • Manual steward review: reserved for conflicts a rule cannot safely auto-resolve — materially different DOBs, conflicting sex/gender fields, or any conflict flagged by a downstream clinical safety report.
Field-level survivorship configuration is itself a governance artifact, typically owned jointly by health information management, clinical informatics, and compliance stakeholders — not left to default vendor behavior.
Every surviving value in a golden record must remain traceable back to the specific source fragment, timestamp, and rule that produced it. This provenance trail serves three distinct purposes:
• Clinical trust: a clinician questioning why a chart shows a particular medication can trace it to the originating encounter and source system rather than treating the merged record as an unexplained black box • Reversibility: if a merge is later discovered to be a false positive, the reconciliation engine must be able to un-merge cleanly — restoring each source fragment to its pre-merge state without losing any documentation added after the erroneous merge occurred • Audit and regulatory response: HIPAA and state health-information-exchange audit requirements frequently require demonstrating exactly which source contributed which data element to a disclosed record
Because of this, production-grade reconciliation systems never physically delete or overwrite source fragments on merge — the golden record is a computed view layered on top of an immutable fragment history, and survivorship decisions are themselves versioned and re-evaluated whenever new source data or corrected identifiers arrive.
A wrong-patient merge that goes undetected can propagate for months before a clinician notices an inconsistency — commonly an allergy or medication that does not match the patient in front of them — making full reversibility, not just merge accuracy, a core clinical safety requirement.
Records flagged as possible-matches by probabilistic scoring, or as unresolved conflicts by survivorship rules, are routed to a human data-steward queue rather than resolved automatically. A typical steward workflow includes:
• Side-by-side comparison view: the candidate fragments displayed field-by-field with agreement/disagreement highlighted, alongside each field's source system and capture date • Confidence context: the probabilistic match score and which specific fields drove it, so the steward understands why the system flagged the pair rather than re-deriving the judgment from scratch • Disposition options: confirm merge, reject as distinct patients, defer pending additional information (e.g., contacting the patient or source facility to resolve a genuine ambiguity) • Downstream audit trail: every steward decision is logged with the reviewer identity and rationale, feeding both the immediate golden record and the longer-run tuning of match thresholds
Large integrated delivery networks commonly see several hundred flagged conflicts enter this queue monthly; queue backlog itself becomes an operational metric, since an unreviewed possible-match sits in limbo — neither confirmed into the golden record nor cleared as distinct.
Reconciliation is not a one-time project — it is a continuous governance discipline. Once a golden longitudinal record is assembled and validated, the Master Patient Index requires ongoing monitoring, periodic re-audit, and increasingly, participation in network-level identity resolution as health information exchange expands beyond any single organization.
Before a reconciled longitudinal record is treated as authoritative for clinical use, it typically passes through a validation gate distinct from the matching and survivorship logic that produced it:
• Structural completeness checks: required identity fields present and internally consistent (DOB not in the future, sex and clinical fields not contradictory) • Clinical safety spot-checks: allergy and medication lists reviewed for plausible duplication or contradictory entries that survivorship rules may have handled imperfectly • Sample-based manual audit: a statistically sampled subset of automatic merges re-reviewed by a data steward, independent of the automatic decision, to estimate real-world false-merge rate rather than relying solely on the algorithm's confidence score • Patient-facing verification, where available: portal-based identity confirmation or registration re-verification can catch a residual false merge that internal review missed
Only after passing validation does a golden record propagate outward to downstream consumers — clinical decision support, population health analytics, and any external HIE or TEFCA-facing interfaces — since errors that reach those consumers are substantially harder to correct retroactively.
A validated MPI degrades if left unmonitored: new source systems onboard, existing systems change their identifier schemes, and patient demographics naturally drift (address changes, name changes, insurance churn) faster than any static rule set anticipated. Mature MPI governance treats duplicate rate as a tracked operational metric, not a one-time cleanup outcome:
• Baseline measurement: industry-cited duplicate rates commonly run 8–12% within a single hospital EHR before active reconciliation, and considerably higher — often exceeding 20% — across federated multi-organization data without a shared identity-resolution layer • Trend monitoring: duplicate rate re-measured on a recurring cadence (commonly quarterly) using sampled audits, since a rising trend signals either an onboarding gap (a new source system feeding unreconciled records) or match-rule drift • Governance ownership: a standing data-governance body — spanning health information management, clinical informatics, IT, and compliance — owns threshold tuning, survivorship rule changes, and escalation of persistent false-merge patterns
Duplicate-rate monitoring is as much an organizational commitment as a technical one: the algorithms in Stages 2–4 only stay effective if governance keeps pace with how source data actually changes.
Industry benchmarks commonly cited for patient matching place accuracy for well-tuned probabilistic systems in the 90–95% range within a single organization, dropping meaningfully at multi-organization scale absent a shared identity-resolution layer — underscoring why network-level identity infrastructure, not just better local algorithms, is required to close the remaining gap.
As health information exchange scales beyond a single organization's MPI, patient identity resolution becomes a network-level problem. The Trusted Exchange Framework and Common Agreement (TEFCA) establishes Qualified Health Information Networks (QHINs) as the connective layer between participating health systems, and patient matching across QHIN boundaries inherits every challenge described in this simulation — at larger scale and with less shared context between organizations that may have never previously exchanged data about a given patient.
Network-level identity resolution approaches generally combine:
• Federated query patterns, where a patient-matching request is broadcast across participating QHINs rather than relying on a single centralized master index • Shared demographic-matching algorithms and minimum-confidence standards, so that a 'match' asserted by one QHIN participant carries comparable meaning to another • Patient-mediated identity assertion, increasingly explored as a complement to algorithmic matching — allowing patients themselves to confirm or correct linkage through patient portals
The long-run trajectory of longitudinal record reconciliation is toward exactly this kind of network-level infrastructure: today's single-organization EMPI remains necessary, but increasingly represents one node feeding into a broader, standards-governed identity-resolution fabric spanning the full care continuum.