Comparative Effectiveness Research on real-world claims & EHR data — from OMOP CDM assembly through propensity-score matching to regulatory-grade effect estimates
Comparative Effectiveness Research does not begin with statistics — it begins with data engineering. Administrative claims (professional, facility, pharmacy), electronic health records, and disease registries are structurally incompatible with each other: different code sets, different visit granularity, different latency. The OHDSI OMOP Common Data Model (CDM) is the industry-standard harmonization layer that makes federated, reproducible analysis across these sources possible.
CER draws on structurally distinct data streams, each with characteristic strengths and blind spots:
Administrative claims (IBM MarketScan, Optum Clinformatics, CMS Medicare 100% files): • Captures every reimbursed encounter — near-complete longitudinal utilization for insured members • Pharmacy claims give objective fill dates and days-supply (better exposure ascertainment than EHR-recorded prescriptions) • Blind spots: no lab values, no vitals, no cause-specific death detail, loses patients on disenrollment
Electronic health records (Flatiron Health oncology mart, TriNetX, Optum EHR, VA CDW): • Rich clinical detail: labs, vitals, ECOG performance status, biomarker/genomic results, physician notes (curated via NLP + human abstraction at Flatiron) • Blind spots: care fragmentation — a patient seen outside the network is invisible; mortality often under-captured without linkage to the National Death Index
Hybrid/linked data (claims-EHR linkage, e.g. Optum linked or Komodo Health): • Combines utilization completeness with clinical granularity • Enables adjustment for confounders (labs, biomarkers) usually only available in charts
Registries (ASCO CancerLinQ, STS National Database, ACC NCDR): • Purpose-built for a disease area; high data quality but limited generalizability and smaller N
All four are ingested into the OHDSI OMOP CDM: source codes mapped to standard concepts via the ATHENA vocabulary browser, populating the PERSON, DRUG_EXPOSURE, CONDITION_OCCURRENCE, MEASUREMENT and OBSERVATION_PERIOD tables. Standardization lets the same analytic cohort/characterization code (via ATLAS or the Book of OHDSI tooling) run unmodified across a 3,500-site federated network — a study run once, replicated everywhere.
Regulators do not accept RWD at face value — the FDA's 2018 Framework for FDA's Real-World Evidence Program and the companion "Data Standards for Drug and Biological Product Submissions Containing Real-World Data" guidance require documented fitness-for-use before RWE can support a labeling claim:
• Reliability: completeness, accuracy, and provenance of the underlying data — was diagnosis coding validated against chart review (positive predictive value typically reported, e.g. PPV >80% for a claims-based algorithm)? • Relevance: does the data source capture the population, exposure, and outcome of interest with adequate follow-up? • Sentinel Initiative: FDA's own active-surveillance CDM-based system spans >100M lives across claims and EHR partners, used routinely for post-market safety signal evaluation and increasingly for effectiveness questions • EMA's parallel infrastructure is DARWIN EU (Data Analysis and Real World Interrogation Network), federated across >20 European data partners on the OMOP CDM • GRACE checklist (Good ReseArch for Comparative Effectiveness) and the RECORD-PE reporting extension operationalize what a well-documented RWD study looks like for peer review and regulatory dossiers
The single most important discipline in CER is specifying the study before touching outcome data: Hernán & Robins' target trial emulation framework forces the analyst to write a full RCT protocol — eligibility criteria, treatment strategies, assignment procedure, follow-up start, outcome, and analysis plan — and only then ask how each element is emulated in observational data. This prevents the two classic observational fallacies: immortal time bias and prevalent-user bias.
Three design choices anchor a defensible comparative effectiveness cohort:
1. New-user (incident-user) design: only patients initiating the drug of interest with no exposure in a washout window (commonly 180–365 days) are eligible. This aligns time-zero with the start of an RCT and avoids "healthy survivor" contamination from prevalent, already-tolerating users.
2. Active-comparator design: instead of comparing drug users to non-users (who differ systematically in health-seeking behavior — the classic "healthy user bias"), the comparator is a clinically equivalent alternative therapy for the same indication (e.g., an SGLT2 inhibitor vs. a DPP-4 inhibitor as second-line add-on therapy in type 2 diabetes, rather than vs. no treatment).
3. Time-zero alignment: eligibility assessment, exposure assignment, and start of follow-up must all occur at the same index date. Violating this — e.g., conditioning on a future event to classify baseline exposure — introduces immortal time bias, which can spuriously exaggerate a survival benefit by construction, independent of the true drug effect.
Grace period and intention-to-treat vs. per-protocol strategies are pre-specified: treatment-policy estimands follow patients regardless of discontinuation/switching (mirrors ITT), while per-protocol estimands censor at first deviation, requiring inverse-probability-of-censoring weights to handle the informative censoring this introduces.
Every operational definition in the emulated protocol needs an auditable, ideally validated, computable phenotype:
• Exposure algorithms: drug_exposure records from pharmacy claims (fill date + days-supply) or e-prescribing; a "current use," "as-treated," or "ever-use" exposure definition is pre-specified with explicit grace-period rules for permissible gaps between refills • Outcome algorithms: e.g., myocardial infarction defined by a validated ICD-10-CM claims algorithm with published PPV against chart-confirmed cases (commonly 90-95% for MI, lower for softer endpoints like heart failure hospitalization) • Covariate assessment window: baseline covariates measured in a fixed look-back period (typically 365 days) strictly prior to index date, never after — post-index covariates measured after treatment initiation are potential mediators and must be excluded from the confounder set to avoid collider bias • Censoring events: disenrollment, death, end of study period, outcome occurrence, or protocol deviation (per estimand) — competing risks (e.g., death precluding a non-fatal outcome) handled via Fine-Gray subdistribution hazard models or cause-specific Cox models depending on the estimand
Because treatment is not randomized in RWD, treated and comparator patients differ systematically at baseline — sicker patients are more likely to receive an add-on therapy, a phenomenon called confounding by indication. The propensity score — the probability of receiving treatment given measured covariates — collapses a high-dimensional confounder space into a single balancing score, enabling matching, stratification, or weighting to emulate randomization on observables.
The propensity score e(X) = P(Treatment=1 | X) is typically estimated by logistic regression (or gradient-boosted trees / LASSO for high-dimensional covariate sets) on pre-index covariates: demographics, comorbidity indices (Charlson, Elixhauser), prior drug exposures, prior utilization intensity (proxy for frailty/health-seeking behavior), and calendar time.
High-dimensional propensity score (hdPS, Schneeweiss et al. 2009) automates confounder discovery: it empirically mines hundreds to thousands of claims codes (diagnoses, procedures, drug classes) ranked by their prevalence and estimated bias impact, selecting the top ~500 as covariates without requiring the analyst to pre-specify every confounder — invaluable for capturing unmeasured-but-coded proxies for disease severity and frailty.
Four uses of the fitted score: 1. 1:1 or 1:k nearest-neighbor matching within a caliper (commonly 0.1–0.2 pooled SD of the logit PS) — unmatched patients are excluded from the analytic cohort 2. Stratification into PS quintiles/deciles, with effect estimated within strata and pooled 3. Inverse probability of treatment weighting (IPTW), creating a pseudo-population where treatment is independent of measured covariates — average treatment effect (ATE) vs. average treatment effect on the treated (ATT, using matching weights) target different estimands 4. Covariate adjustment — the PS entered as a single regressor in the outcome model, the weakest of the four approaches for confounding control
A frequently cited empirical benchmark: Franklin et al. (2017, Epidemiology) replicated 21 completed RCTs using claims-based PS-matched cohorts and found that when new-user, active-comparator design and rigorous PS matching were used, RWD estimates agreed with RCT point estimates in direction and approximate magnitude for the large majority of comparisons — but design choices (not statistical technique) explained nearly all of the discordant cases.
Matching success is judged on covariate balance, not on statistical significance of baseline differences (which is sample-size dependent and misleading). The standardized mean difference (SMD) is the balance metric of record:
SMD = (mean_treated − mean_comparator) / pooled_SD
Convention (Austin, 2009): SMD < 0.10 indicates good balance; 0.10–0.25 marginal; >0.25 meaningful residual imbalance requiring re-specification of the PS model (added interaction terms, splines for continuous covariates, or a tighter caliper).
A love plot — a dot plot of each covariate's SMD before and after matching, covariates sorted by magnitude — is the standard visual QC artifact accompanying every PS-matched CER publication and regulatory submission. Positivity (overlap) is checked via PS distribution histograms by arm: regions of the covariate space where only one arm is represented (e.g., PS > 0.9 populated only by treated patients) violate the positivity assumption underlying causal identification and those patients must be trimmed rather than extrapolated over.
With a balanced, matched cohort in hand, the outcome analysis itself is comparatively straightforward — the hard causal-inference work already happened upstream in design and matching. Time-to-event outcomes (death, hospitalization, disease progression) are analyzed with Kaplan-Meier estimation and Cox proportional-hazards regression stratified on matched pairs, with intention-to-treat and per-protocol estimators run side by side per the pre-specified estimand.
Kaplan-Meier curves are generated separately for the treated and comparator arms of the matched cohort, non-parametrically estimating the survival function S(t) = P(T>t) as a step function that drops at each observed event time, accounting for right-censoring (disenrollment, end of study, competing death). The vertical separation between the two curves over follow-up time is the raw, unadjusted visual signature of a treatment effect surviving the matching step.
The formal effect estimate comes from a Cox proportional-hazards model stratified on matched pair (or clustered with robust sandwich variance for 1:k matching), which conditions out any residual within-pair confounding and yields:
h(t | treatment) = h₀(t) × exp(β × Treatment)
HR = exp(β), interpreted as the instantaneous rate ratio of the event in the treated vs. comparator arm, assumed constant over follow-up (the proportional-hazards assumption, checked via Schoenfeld residuals or log-log survival plots — time-varying hazard ratios call for a flexible alternative like restricted mean survival time (RMST) or spline-based hazard models).
Competing risks: when death precludes observation of a non-fatal outcome (e.g., disease progression), a cause-specific hazard model answers an etiologic question while a Fine-Gray subdistribution hazard model answers a prognostic/absolute-risk question — the two can point in different directions and the estimand must specify which is intended.
CER studies increasingly report both estimators side by side, mapped explicitly onto the ICH E9(R1) estimand framework:
• Treatment-policy estimand (ITT-analog): patients are analyzed according to their initially assigned exposure regardless of subsequent discontinuation, switching, or added therapy. This estimates the effect of the initial prescribing decision — most relevant to a formulary or first-line-choice question — but can dilute a true per-exposure effect if substantial crossover occurs.
• Per-protocol estimand: follow-up is censored at the first treatment deviation (discontinuation, switch, or added comparator drug). Because censoring here is likely informative — sicker patients disproportionately discontinue — naive per-protocol analysis is biased; the standard correction is inverse-probability-of-censoring weighting (IPCW), using a second, time-varying model of the probability of remaining on protocol given evolving covariates.
In the case simulated here, ITT gives HR 0.79 (0.71–0.88) while the IPCW-adjusted per-protocol estimate strengthens to HR 0.74, consistent with dilution from non-adherent, informatively-censored patients in the ITT population — a pattern regulators specifically look for as a marker of a plausible, internally consistent effect rather than an analytic artifact.
No observational adjustment eliminates the risk of unmeasured confounding — the honest final step of CER is quantifying how fragile the result is. Negative-control outcome calibration, quantitative bias analysis, and the E-value give regulators and payers a defensible answer to "how much hidden bias would it take to explain this away?" before the evidence supports a label expansion or coverage decision.
A negative control outcome is an endpoint with no plausible causal relationship to the treatment (e.g., ingrown toenail, appendicitis) that is nonetheless subject to the same measured and unmeasured confounding structure as the true outcome of interest. If the fully-adjusted analysis shows a non-null association with a negative control (e.g., HR 1.3 for an outcome that cannot be caused by the drug), this quantifies residual systematic bias in the study design/analysis pipeline.
Empirical calibration (Schuemie et al., OHDSI methodology) fits a null distribution across dozens of negative control outcomes and uses it to calibrate the p-value and confidence interval of the outcome of interest — routinely widening nominal 95% CIs by 20-40% once systematic error, not just random error, is accounted for. Negative control exposures (drugs known not to affect the outcome) serve a parallel diagnostic role. This calibration step is now expected practice in FDA Sentinel active-surveillance analyses and increasingly in RWE submissions.
The E-value (VanderWeele & Ding, Annals of Internal Medicine, 2017) answers a single, interpretable question: on the risk-ratio scale, how strongly would an unmeasured confounder need to be associated with both treatment and outcome, above and beyond all measured covariates already adjusted for, to fully explain away the observed association?
For an observed hazard ratio (approximated as a risk ratio for a rare-ish outcome) of 0.79, the E-value is calculated on the transformed scale and equals approximately 1.94: an unmeasured confounder would need to be associated with roughly a 1.94-fold increase in both treatment initiation and outcome risk, independent of measured covariates, to nullify the finding. The second E-value, calculated at the CI bound closer to the null (1.61), gives the more conservative "how much bias to move the CI to include the null" answer.
E-values in the 1.5–2.5 range are typically judged only moderately robust — plausible for confounders like unmeasured frailty or performance status in oncology CER; E-values >4 are considered strong evidence that no single unmeasured factor plausibly explains the result, absent an implausibly strong hidden confounder.
Pfizer's 2019 FDA approval of palbociclib (Ibrance) for male breast cancer is the frequently cited precedent for RWE-substantiated oncology label expansion: because a randomized trial in this rare population (<1% of breast cancer cases) was infeasible, real-world data from Flatiron Health and IQVIA claims/EHR sources — supplemented by post-marketing pharmacovigilance case reports — supported the expanded indication, with FDA explicitly citing the RWE submission in the approval letter as adequate evidence of effectiveness in this population.
A completed CER study feeds into distinct downstream evidentiary pathways, each with its own standards body:
• FDA: the 2018 RWE Framework (enabled by the 21st Century Cures Act, 2016) governs use of RWE to support new indications for approved drugs and to satisfy post-approval study requirements; Sentinel supplies the safety-surveillance counterpart. STaRT-RWE and the ICH E2B(R3)/E2E pharmacovigilance data standards structure submission format. • EMA: DARWIN EU and GVP Module VIII (Post-Authorisation Safety Studies) plus the ENCePP Code of Conduct set methodological and transparency expectations; a registered ENCePP study seal signals pre-specification and protects against post-hoc outcome switching. • Health Technology Assessment / payers: ICER (Institute for Clinical and Economic Review) in the US and NICE in the UK increasingly accept well-conducted RWE cost-effectiveness analyses, and CMS Coverage with Evidence Development links reimbursement to a prospective RWE data-collection commitment. • Reporting standards: the RECORD-PE extension to STROBE, and the ISPOR-ISPE joint task force good-practice guidance, define the checklist reviewers use to judge whether pre-registration, cohort definitions, and sensitivity analyses meet the bar for regulatory or HTA reliance — pre-registration on a public protocol repository (e.g., EU PAS Register, ENCePP) before outcome data are examined is now close to a hard requirement for FDA-facing RWE.