Government Record Linkage — Probabilistic Entity Resolution

Dataset

Records per agency 24

Field error rate 15%

Decision Rule

Match threshold T 15.0

Declared match — correct pair
Declared match — false pair

Linkage Results

Precision

—

Recall

—

Matches

—

True positives

—

False positives

—

How it works

This is the Fellegi–Sunter model of probabilistic record linkage — the mathematics behind how governments and hospitals deduplicate citizen records that live in separate databases with no shared key.

Every candidate pair (one record from Agency A, one from Agency B) is compared field by field. For each field i we know two probabilities from historical audits:

m_i = P(fields agree | the two records are the same person)
u_i = P(fields agree | the two records are different people, by chance)

Each field contributes a log-likelihood weight to a total match score R:

agree on field i:     + log2( m_i / u_i )
disagree on field i:  + log2( (1-m_i) / (1-u_i) )
R = sum of all four field weights

A highly discriminative field (rare to agree by chance, e.g. date of birth or an ID number) swings R hard when it agrees. A weak field (common to share by chance, e.g. a shared street address) barely moves R.

Field	m (agree if same)	u (agree by chance)
Name	0.90	~0.003
Date of birth	0.97	~0.00005
Street address	0.75	~0.083
4-digit ID suffix	0.90	0.0001

Declare a match when R ≥ threshold T. Raise T and you demand stronger evidence — fewer false links but more true pairs missed (lower recall). Lower T and you catch more true pairs but let in coincidental look-alikes (lower precision). Raising the field error rate simulates messier government data entry (typos, address changes between agency updates) — it pushes true pairs' scores down, since fewer fields agree even for the same citizen.

Drag to orbit the scene: Agency A records sit on the left plane, Agency B on the right — deliberately shuffled, since in real siloed databases there is no shared ordering to exploit. Green links are correct matches at the current threshold; red links are false matches — data errors that coincidentally scored high enough.

AGENCY AAGENCY B

Agency A record Agency B record

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.