Automated completeness, consistency, accuracy, timeliness, and OMOP CDM conformance scoring at each site, aggregated without centralizing raw records — gating and weighting federated pool contribution
Before a site's data can meaningfully contribute to a federated health dataset or federated model training run, it must pass a standardized, automated quality assessment computed entirely within that site's own infrastructure. Five dimensions dominate practical clinical data quality frameworks: completeness (how much required data is actually present), consistency (do related fields agree with each other), accuracy (do values fall within clinically plausible ranges), timeliness (how current is the data), and conformance to a common data model — most commonly OMOP CDM in multi-institutional research networks.
1. Completeness — is the data actually there? • Field-level missingness rate: fraction of records with a null/absent value for each required field • Structural completeness: are entire expected tables/domains present (e.g., a site with no procedure records at all is a structural, not just field-level, gap) • Computed as: completeness_score = 1 − (missing_required_fields / total_required_fields), aggregated across records
2. Consistency — do related fields logically agree? • Cross-field validation rules: e.g., discharge_date must be ≥ admission_date; a male-coded patient should not have a pregnancy-related diagnosis code • Referential integrity: every diagnosis record should reference a valid, existing encounter record • Temporal consistency: medication start dates should precede or coincide with, not follow, the associated prescription order
3. Accuracy (plausibility) — are values clinically believable? • Range checks: lab values, vital signs compared against physiologically plausible bounds (e.g., adult heart rate 20–300 bpm as an outer plausibility bound, not a clinical normal range) • Uniqueness checks: duplicate patient/encounter records inflating apparent cohort size • Kahn et al. (2016) formalized this as "plausibility" — distinguishing values that are merely present and internally consistent from values that are actually credible given clinical knowledge
4. Timeliness — how current and how promptly updated is the data? • Latency: time elapsed between a clinical event occurring and its corresponding record becoming available in the site's data warehouse • Currency: how recently was the extract refreshed relative to the federated training/analysis run • Sites with long ETL (extract-transform-load) lag can silently bias a federated model toward stale clinical practice patterns
5. Conformance to a common data model (OMOP CDM): • OMOP CDM (OHDSI's Observational Medical Outcomes Partnership Common Data Model) standardizes clinical concepts to shared vocabularies (SNOMED, RxNorm, LOINC) across otherwise heterogeneous source EHR systems • Conformance scoring checks structural conformance (correct tables/fields populated per the CDM specification) and terminology conformance (source codes correctly mapped to standard OMOP concept IDs, not left as unmapped/orphan codes) • High unmapped-concept rates are a leading indicator that a site's ETL pipeline needs remediation before its data is trustworthy for pooled or federated analysis
Once each site has computed its local quality scores, those scores — not any underlying patient record — are transmitted to a central aggregator. This mirrors exactly the privacy-preserving pattern used in federated model training itself (see the companion cross-institutional federated learning simulation): only summary statistics cross the institutional boundary, giving the consortium a network-wide quality dashboard while every site retains full control over its raw data.
Composite score construction: • Each dimension (completeness, consistency, accuracy, timeliness, conformance) is normalized to a common 0–100 scale • A weighted sum produces the composite: Q_k = Σ w_d · dimension_d, where weights w_d reflect the consortium's governance priorities (e.g., a research network focused on medication safety may weight medication-domain completeness more heavily than a network focused on imaging) • Weights are agreed upon by consortium governance BEFORE scoring begins, not adjusted retroactively per site — this prevents the scoring criteria from being tuned to favor or penalize any particular participant after the fact
Why aggregating scores (not raw distributions) preserves privacy: • A raw distribution histogram of, say, lab values can itself leak information about small subpopulations at a site (e.g., a site with only 3 patients with a rare condition would have a histogram that is nearly re-identifying for those patients) • Reporting only a single composite quality number, or coarse per-dimension sub-scores, avoids this leakage surface while still giving the aggregator everything it needs to make admission/weighting decisions • Sites requiring stronger guarantees can add differential-privacy noise to their reported scores before transmission, exactly analogous to DP-SGD noise addition during federated model training
The network quality dashboard: • Central aggregator maintains a live view of every participating site's current composite score, per-dimension breakdown, and score trend over time • Used by consortium governance (not any single site) to make network-wide decisions: which sites are eligible for a given study, how much weight each contributes to a federated model (Stage 4), and where remediation outreach should be prioritized • Because scores — not raw data — are aggregated, a new site can be evaluated for consortium admission by simply running the standardized local scoring module and reporting results, without a lengthy raw-data pre-assessment process
Aggregate quality scores can mask localized problems: a site with a good overall completeness rate can still have a cluster of duplicated records, a batch of implausible lab values from a miscalibrated analyzer, or a subpopulation with systematically corrupted timestamps. Statistical and machine-learning anomaly detectors run at the record and distribution level catch these localized issues before they can silently degrade a pooled dataset or bias a federated training run.
Statistical outlier detection: • Z-score / modified z-score: flags individual values more than k standard deviations (typically k=3, or using median absolute deviation for robustness to the outliers themselves) from the site's local distribution mean — catches gross data-entry errors and unit-conversion mistakes (e.g., a weight recorded in pounds where kilograms was expected) • IQR (interquartile range) fencing: flags values outside Q1 − 1.5·IQR or Q3 + 1.5·IQR — more robust than z-score for skewed clinical distributions (most lab values are not normally distributed)
ML-based anomaly detection: • Isolation forest: builds random partitioning trees where anomalous points (which differ from the bulk of the data on at least one feature) get isolated in fewer splits than normal points — efficient for high-dimensional structured EHR feature vectors where simple univariate rules miss multivariate anomalies • Autoencoder reconstruction error: a neural autoencoder trained to compress-and-reconstruct normal records will have high reconstruction error specifically on records unlike anything in its training distribution — useful for catching subtle multivariate anomalies (a combination of values each individually plausible but jointly implausible)
Duplicate and near-duplicate detection: • Fuzzy/probabilistic record linkage (e.g., Fellegi-Sunter model) identifies likely duplicate patient records across name/DOB/address variants even when no field matches exactly — critical because duplicate patients silently inflate apparent cohort size and can double-count outcomes
Distributional drift detection: • Each new data cut's feature distributions are compared against a rolling baseline (e.g., population stability index, or KS-test per feature) to detect systemic shifts — a sudden jump in missingness for a specific lab, or a shift in mean value, often signals an upstream EHR configuration change, a new equipment vendor, or an ETL pipeline bug rather than genuine population change • Drift alerts are routed back to the originating site for investigation before the affected data cut is admitted to the federated pool, rather than being silently absorbed into training data
Why this runs BEFORE pool inclusion, not after: • Catching a corrupted cohort after it has already influenced several rounds of federated model training requires re-training from a checkpoint before the contamination — far more costly than a pre-admission screening pass that simply excludes or flags the problem data upfront
A simple pass/fail quality gate is a blunt instrument: it discards a site's entire contribution even if only some dimensions are weak, and it treats a site that barely passes identically to one that scores near-perfect. A more refined approach folds the composite quality score directly into the federated averaging weight, so every admitted site contributes to the global model in proportion to both its data volume AND its data quality — down-weighting noisy or incomplete sites rather than excluding them outright.
Quality-weighted FedAvg:
Standard FedAvg (size-only weighting): w_{t+1} = Σ_k (n_k / n) · w_k
Quality-weighted FedAvg: w_{t+1} = Σ_k (q_k · n_k / Σ_j q_j · n_j) · w_k
where q_k ∈ [0,1] is site k's normalized composite quality score (from Stage 2's aggregation), rescaling each site's effective contribution by both how much data it has AND how trustworthy that data is.
Why this matters in practice: • A large site with mediocre data quality (say, q_k=0.5, n_k=50,000 records) and a smaller site with excellent quality (q_k=0.95, n_k=15,000 records) contribute more comparably under quality-weighting than under size-only FedAvg, which would let the larger-but-noisier site dominate the global model • Prevents a scenario where a single high-volume but poorly-instrumented site (e.g., one still mid-migration to a new EHR, with elevated missingness) disproportionately drags the global model toward its own data artifacts simply because it has more raw records • Still respects a hard governance floor: sites below a minimum quality threshold (this simulation's adjustable gate threshold) are excluded entirely rather than merely down-weighted — quality weighting refines contribution among admitted sites, it does not replace the admission gate itself
Dynamic re-weighting: • Because quality scores are re-computed at every new data cut (Stage 5), a site's contribution weight can rise as it remediates known issues (e.g., completes an EHR migration, fixes a mapping bug) or fall if new problems emerge — the federated training pipeline adapts automatically rather than requiring a manual consortium vote each time • Transparency matters for consortium trust: each site should be able to see its own quality score breakdown and resulting training weight, so quality improvement is an actionable, visible incentive rather than an opaque penalty imposed from the center
Data quality is not static: EHR system migrations, new equipment vendors, staffing changes, and coding-practice drift can degrade a previously high-quality site's data over time, just as remediation efforts can improve a previously marginal site. Treating quality scoring as a continuous monitoring program — re-scored at every new data cut, with drift alerts and a governance audit trail — is what keeps a federated health dataset trustworthy over its operational lifetime rather than only at initial onboarding.
Why quality drifts over time, even at previously reliable sites:
• EHR system migrations: a site switching EHR vendors (e.g., transitioning to a new Epic/Cerner instance) frequently sees a temporary spike in missingness and mapping errors during the cutover period, even though the underlying clinical care is unchanged • New equipment/instrumentation: a new lab analyzer or imaging scanner can shift measurement distributions and units in ways that look like a data-quality anomaly if not accounted for in the accuracy/plausibility scoring • Staffing and workflow changes: turnover in clinical documentation staff or coding personnel changes the thoroughness and specificity of diagnosis/procedure coding, directly affecting completeness and conformance scores • Policy or reimbursement changes: shifts in what must be documented for billing purposes can inflate or deflate the apparent prevalence of certain diagnosis codes independent of true clinical incidence
Continuous monitoring program components: 1. Scheduled re-scoring: every new data cut (typically monthly or quarterly, aligned to the federated training/analysis cadence) triggers a fresh run of the full quality-scoring module at each site — not a one-time onboarding certification that is assumed to remain valid indefinitely 2. Drift alerting: automated comparison of the new score against each site's own historical baseline (not just against the consortium-wide threshold) flags sites whose quality has moved meaningfully, in either direction, triggering a lightweight investigation before the next federated training round uses that data cut 3. Governance audit trail: every admission decision, weighting calculation, and threshold exception is logged immutably, giving the consortium (and any external auditor or regulator) a complete, reconstructable history of why any given federated model was trained on the data mix it was trained on 4. Periodic policy reassessment: the consortium governance board revisits the quality dimension weights, gate threshold, and scoring methodology itself on a regular cycle (not per-site, but network-wide) — because what counts as "good enough" quality can reasonably evolve as the consortium's scientific goals, regulatory environment, or member composition changes
The cultural dimension: • A punitive, opaque quality-gating process discourages sites from participating or from being forthcoming about known data limitations • A transparent, actionable scoring program — where sites see their own breakdown, understand exactly what drove a score change, and have a clear remediation path back to full weighting — turns quality scoring into a collaborative improvement mechanism for the whole consortium rather than an adversarial pass/fail checkpoint
The most common real-world failure mode in federated health data networks is not a single catastrophic data-quality incident, but slow, silent drift at one or two sites going undetected between infrequent manual audits — which is exactly why continuous automated re-scoring, not periodic manual review, is the practice that consortia like OHDSI's OMOP network have converged on as standard governance.