🌐 Synthetic Data Generator Utility-Privacy Tradeoff

From EHR Tables to a Trainable Feature Matrix — and Why De-identification Alone Isn't Enough

Before any generative model can be trained, the real cohort must be assembled into a structured, mixed-type feature matrix from heterogeneous clinical source systems — diagnoses coded in ICD-10, labs in LOINC, medications in RxNorm/NDC, and unstructured demographics. Traditional de-identification (removing the 18 HIPAA Safe Harbor identifiers) reduces but does not eliminate re-identification risk, because rare combinations of quasi-identifiers (age, ZIP3, rare diagnosis, visit date pattern) can still uniquely fingerprint an individual when linked against external data. This is the core motivation for generating wholly synthetic records instead of releasing scrubbed originals.

• 87%: Sweeney re-identification (of US pop. unique on ZIP+DOB+sex)
• 18: HIPAA Safe Harbor fields (identifiers removed, insufficient alone)
• 500–5,000: Typical EHR feature dim. (diagnoses+labs+meds one-hot/binned)
• ICD-10, LOINC, RxNorm: Common code systems (diagnosis, lab, medication coding)

Why re-identification risk survives naive de-identification

Latanya Sweeney's foundational 1997/2000 work showed that 87% of the US population can be uniquely identified from just three quasi-identifiers: 5-digit ZIP code, date of birth, and sex — none of which are HIPAA "direct identifiers," yet in combination they act as a fingerprint. Applied to clinical data:

Quasi-identifier linkage attack: • Adversary obtains a "de-identified" clinical extract (ZIP3, age, sex, rare diagnosis, admission month retained for research utility) • Adversary cross-references a public/leaked voter-registration or commercial data file containing name + ZIP + DOB • Intersection of rare-diagnosis + demographic combination re-identifies specific patients with high confidence

Why generative synthesis is the answer researchers increasingly reach for: • A generative model trained on the real cohort learns the joint distribution P(diagnoses, labs, meds, demographics) but does not memorize a lookup table mapping demographics→identity • Every synthetic record is a novel sample from the learned distribution — in principle no synthetic row corresponds to any single real patient • "In principle" is the operative phrase: overfit generators can still leak near-copies of rare/outlier training records — precisely what Stage 5's membership-inference and nearest-neighbor tests are built to detect

Feature matrix construction (typical pipeline): • Continuous: lab values (e.g., HbA1c, creatinine), binned or z-scored per encounter • Categorical: ICD-10 diagnosis codes (one-hot or hierarchical grouping via CCS/PheWAS categories to control dimensionality) • Discrete/count: medication order counts, encounter counts per period • Demographics: age band, sex, coarse geography (state, not ZIP) — deliberately coarsened to reduce residual re-identification surface even before synthesis begins

CTGAN, medGAN, and Tabular Diffusion — Three Architectures for Learning Mixed-Type Clinical Distributions

Health records are neither pure images nor pure text — they are mixed-type tables combining continuous labs, multi-modal distributions, sparse high-cardinality categorical codes, and structural correlations (a diagnosis implies certain labs are more likely ordered). Three generative architectures dominate synthetic EHR/claims literature: CTGAN's conditional GAN with mode-specific normalization, medGAN's autoencoder-GAN hybrid tuned for sparse binary code data, and diffusion-based tabular generators that iteratively denoise toward realistic rows.

• 2019: CTGAN publication (Xu et al., NeurIPS)
• 2017: medGAN publication (Choi et al., MLHC)
• 2022–23: Tabular diffusion (TabDDPM) (Kotelnikov et al.)
• 300–1,000: Typical training epochs (GAN; diffusion needs 1k+ denoise steps)

Architecture mechanics — conditional normalization, autoencoding, and denoising

CTGAN (Conditional Tabular GAN): • Mode-specific normalization: each continuous column is fit with a variational Gaussian mixture (VGM); a value is represented by (which mode, normalized value within that mode) — handles multi-modal lab distributions (e.g., a lab that clusters at "normal" and "critically abnormal") • Conditional generator: training batches are conditioned on a randomly sampled discrete column+value pair, with a matching real-data sample, to fight class imbalance in rare diagnosis codes • Training-by-sampling: ensures rare categories are seen by the generator proportionally more often than their raw frequency, preventing mode collapse onto only the majority classes • Discriminator (critic): Wasserstein loss with gradient penalty (WGAN-GP) for more stable convergence than vanilla GAN cross-entropy loss

medGAN: • Two-stage: (1) pretrain an autoencoder on sparse binary/count EHR feature vectors (diagnosis/medication indicator vectors) to learn a dense latent representation; (2) train a GAN generator to produce latent vectors, decoded through the frozen autoencoder decoder back to the sparse binary space • Well suited to very high-dimensional sparse code data (thousands of ICD-10/RxNorm indicator columns) where direct GAN generation in raw sparse space is unstable • Minibatch averaging and batch normalization used to combat mode collapse, a persistent GAN failure mode where the generator produces limited diversity

Tabular diffusion (e.g., TabDDPM): • Forward process gradually adds Gaussian noise (continuous columns) or multinomial/categorical noise (discrete columns) to real rows over T steps • A neural network learns to reverse the process, denoising step by step from pure noise back to a realistic row • Generally produces higher fidelity on complex multi-modal joint distributions than GAN baselines in recent benchmarks, at the cost of slower sampling (many denoising steps per row vs. one forward pass for a GAN)

Common failure modes across all three: mode collapse (generator produces a narrow subset of the true distribution), rare-category disappearance (infrequent diagnosis codes never sampled), and correlation collapse (marginal distributions match but joint/conditional structure — e.g., diagnosis-lab correlation — does not).

Differentially Private SGD — Bounding What Any Single Patient Can Change About the Trained Generator

Differential privacy (DP) gives a formal, mathematically provable bound on how much any single training record can influence the model's output distribution. DP-SGD (Abadi et al., 2016) achieves this during generator training by clipping each per-example gradient to a fixed norm and adding calibrated Gaussian noise before the aggregate parameter update — converting an otherwise unbounded per-record influence into a provably bounded one, at the cost of noisier, lower-fidelity gradients.

• 2016: DP-SGD publication (Abadi et al., CCS)
• 1–10: Typical clinical ε range (tighter = stronger guarantee)
• ~1.0: Gradient clip norm C (typical starting hyperparameter)
• Rényi DP / moments accountant: Accounting method (tracks cumulative ε over steps)

The DP-SGD mechanism and the (ε, δ)-differential privacy guarantee

(ε, δ)-Differential Privacy, formally: A randomized training mechanism M satisfies (ε, δ)-DP if for any two datasets D, D′ differing in exactly one record, and any output set S: Pr[M(D) ∈ S] ≤ e^ε · Pr[M(D′) ∈ S] + δ

Interpreted plainly: whether or not any single patient's record was included in training, the probability of observing any particular trained model (or its outputs) changes by at most a factor of e^ε. Smaller ε = a tighter bound = stronger privacy = a proof that no single record could have shifted the model much. δ is a small "failure probability" allowance, typically set below 1/(dataset size).

DP-SGD mechanism, per training step: 1. Per-example gradient computation: for each record i in the minibatch, compute gradient g_i of the loss w.r.t. model parameters — NOT averaged across the batch first 2. Per-example clipping: rescale each g_i so ||g_i||₂ ≤ C (clip norm) — bounds the maximum possible influence any single record's gradient can have on the update, regardless of how extreme an outlier that record is 3. Noise addition: sum the clipped gradients and add Gaussian noise: g̃ = Σg_i + N(0, σ²C²I) 4. Parameter update: θ ← θ − η · g̃ / batch_size, using the noised, clipped aggregate

Privacy accounting (why it is not simply "add noise once"): • Each SGD step leaks a small amount of privacy budget; training runs thousands of steps • Naive composition (summing worst-case ε per step) would make ε explode to uselessness over a full training run • Moments accountant (Abadi et al.) and its successor Rényi DP accounting track the cumulative privacy loss tightly, giving a much smaller final ε for the same noise level than naive composition would suggest — this tighter accounting is what made DP-SGD practical for deep model training at all

The fidelity cost: • Clipping distorts gradient direction for any record whose true gradient exceeds C, disproportionately harming rare/outlier patients — exactly the population whose data patterns (rare diagnoses, unusual lab combinations) are often most clinically interesting • Noise addition directly degrades convergence — smaller ε (more privacy) requires proportionally more noise (σ ∝ 1/ε at fixed δ), which is the direct mechanical source of the utility-privacy tradeoff explored in the next two stages

Measuring Utility — Statistical Fidelity Metrics and Train-Synthetic-Test-Real (TSTR) Evaluation

A synthetic dataset is useful only insofar as it preserves what downstream analysts and ML models actually need — marginal distributions, pairwise correlations, and predictive signal. Two complementary evaluation families are standard: distributional fidelity metrics that compare synthetic to real statistics directly, and task-based TSTR evaluation that trains a real downstream model (e.g., a readmission-risk classifier) on synthetic data and measures how well it performs on real held-out data.

• 0 (identical) – 1: KS statistic range (per-column distribution distance)
• within 0.02–0.05: TSTR AUC target (of TRTR (real-trained) baseline)
• Frobenius norm: Correlation distance metric (ΔCorr matrix, real vs. synthetic)
• 0.80–0.88: Typical fidelity at ε=4 (composite fidelity score)

Fidelity metrics and the TSTR / TRTR protocol

Distributional fidelity metrics:

1. Kolmogorov-Smirnov (KS) statistic (continuous columns): • KS = sup_x |F_real(x) − F_synthetic(x)|, the maximum gap between empirical CDFs • Reported per column and averaged; "fidelity" often reported as 1 − mean(KS) so higher is better • Sensitive to distribution shape mismatches DP noise or generator mode collapse introduce

2. Total Variation Distance / contingency-table divergence (categorical columns): • Compares empirical frequency tables of categorical/diagnosis codes between real and synthetic • Captures whether rare diagnosis codes are under- or over-represented in synthetic output

3. Pairwise correlation matrix distance: • Compute Pearson/Cramér's V correlation matrix on real data and on synthetic data • Frobenius norm of the difference matrix quantifies whether joint relationships (e.g., diabetes diagnosis correlated with elevated HbA1c) survive synthesis — marginal fidelity alone can look good while this joint structure collapses

TSTR (Train on Synthetic, Test on Real) protocol: 1. Train a downstream predictive model (e.g., gradient-boosted tree or logistic regression predicting 30-day readmission) entirely on the synthetic dataset 2. Evaluate that trained model's AUC/AUPRC on a held-out REAL test set that was never seen by the generator or the downstream model 3. Compare against TRTR (Train on Real, Test on Real) — the same downstream model trained directly on real training data — as the achievable ceiling 4. Utility gap = AUC_TRTR − AUC_TSTR; a well-tuned generator at moderate privacy budgets typically holds this gap to 0.02–0.05 AUC; aggressive DP noise (small ε) can widen it substantially

Why TSTR matters more than fidelity metrics alone: • A generator can score well on marginal KS fidelity while still failing TSTR, if it fails to preserve the specific joint feature-outcome relationships a classifier depends on • Regulatory and research use cases (e.g., using synthetic data to prototype a risk model before requesting real-data access) care specifically about TSTR-style task performance, making it the practical bar generator teams are held to

Attacking Your Own Generator — Membership Inference, Nearest-Neighbor Ratios, and Choosing an Operating Point

A privacy budget ε is a theoretical upper bound, not empirical proof that a specific trained generator is actually safe to release. Responsible synthetic-data pipelines run empirical privacy attacks against the trained generator — most importantly membership inference attacks (MIA) and nearest-neighbor distance ratio (NNDR) analysis — and use the resulting utility-privacy Pareto frontier, swept across candidate ε values, to select a defensible operating point rather than picking ε arbitrarily.

• ≤ 0.55: MIA AUC target (0.50 = attacker no better than chance)
• > 0.8–0.9 typical: NNDR safe threshold (ratio of 1st to 2nd nearest-neighbor dist.)
• ε ∈ [0.5, 20]: Pareto sweep range (typical clinical synthesis study range)
• 0.85–0.98: No-DP MIA AUC (baseline) (overfit generators leak strongly)

Membership inference, nearest-neighbor leakage, and reading the Pareto frontier

Membership Inference Attack (MIA): • Threat model: attacker has a candidate record and query access to the generator (or its synthetic outputs); goal is to decide whether that candidate record was part of the real training set • Shadow-model approach (Shokri et al. 2017): attacker trains "shadow" generators on datasets with/without a target-like record, trains a binary meta-classifier on the resulting output statistics (e.g., density of synthetic points near the candidate) to distinguish "member" vs. "non-member" behavior, then applies that meta-classifier to the real target generator • Reported as attack AUC: 0.50 = attacker does no better than a coin flip (strong privacy in practice); 1.00 = attacker perfectly identifies members every time (severe leakage) • Undefended generators (ε=∞, no DP) trained to convergence commonly show MIA AUC of 0.85–0.98 on outlier/rare-pattern records — the generator has effectively memorized and reproduced them

Nearest-Neighbor Distance Ratio (NNDR): • For each real record, compute distance to its nearest synthetic neighbor (d1) and second-nearest synthetic neighbor (d2); ratio = d1/d2 • A ratio near 1 means the two nearest synthetic points are about equally close — no single synthetic record is suspiciously close (i.e., a near-duplicate) to any specific real record • A ratio near 0 means one synthetic record sits dramatically closer than any other — a strong signal of memorization/near-copying of that specific real patient • Applied dataset-wide as a distribution; the fraction of real records with NNDR below a safety threshold is reported as a leakage-risk summary statistic

Reading the Pareto frontier: • Sweeping ε from very small (strong privacy, e.g., ε=0.5) to very large (weak/no privacy, e.g., ε=20+) traces a curve of achievable (utility, privacy) pairs • Points on the frontier are Pareto-optimal: no achievable configuration improves either utility or privacy without worsening the other • Points strictly inside the frontier (achievable but dominated) indicate a poorly tuned generator — more careful architecture/hyperparameter choices could achieve strictly better utility at the same privacy level, or vice versa • Selecting ε is ultimately a governance decision, not a purely technical one: it weighs the sensitivity of the specific cohort (rare disease registries need smaller ε / stronger privacy than large common-condition cohorts), the intended downstream use, and applicable regulatory guidance — the frontier tells you what tradeoffs are achievable, not which one you should choose

A generator that reports strong fidelity metrics (KS, correlation distance) but has never been stress-tested with a membership-inference attack has not actually been evaluated for privacy — fidelity and privacy are measured by entirely different procedures, and a high-fidelity, high-ε ("weak privacy") synthetic release can leak specific patients' records even while looking statistically excellent on every utility metric.