🗂 Data Cleaning & SDTM Mapping Pipeline

Extracting Raw Data from EDC and Working the Query Loop

Every SDTM dataset begins its life as noisy, incomplete, human-entered data inside an Electronic Data Capture (EDC) system — Medidata Rave, Veeva Vault CDMS, or Oracle InForm. Before a single record can be transformed into a standard, it must survive an edit-check gauntlet and a query resolution cycle that can span the entire length of a trial.

• <2%: Typical open queries at DBL (of total data points, per TransCelerate benchmark)
• 800–1,500: Edit checks per Phase III CRF (programmed in EDC build)
• 5–8 days: Median query turnaround (site response time)
• 9: ALCOA+ principles (attributable, legible, contemporaneous...)

EDC build, edit checks, and the discrepancy management lifecycle

Clinical data management begins long before patient 1 is enrolled, with the EDC build:

CRF design and CDASH alignment: • Case Report Forms are designed against CDASH (Clinical Data Acquisition Standards Harmonization) domains so that field-level capture already anticipates downstream SDTM variables • Each EDC field is mapped in a Data Management Plan (DMP) to its eventual SDTM target — this mapping specification is written before first-patient-in, not retrofitted afterward

Edit check taxonomy: • Univariate checks: range checks (e.g., systolic BP 60–260 mmHg), format checks (ISO 8601 partial dates), mandatory-field checks • Cross-form checks: AE start date must fall within study participation window; concomitant medication stop date cannot precede start date • Cross-visit checks: unscheduled visit vital signs compared against prior visit for implausible deltas (>30% weight change flags for query) • External checks: lab values compared against central lab reference ranges pushed via SFTP/API feed, auto-flagging out-of-range results

Discrepancy management workflow: • System-generated query fires automatically when an edit check trips → routed to site coordinator queue in the EDC • Manual query: raised by a Clinical Research Associate (CRA) or Data Manager during Source Data Verification (SDV) or targeted risk-based monitoring review • Query aging report: tracked weekly; queries open >30 days escalate to the sponsor's Data Management Lead • Query closure requires site response + DM adjudication — average of 1.3 query cycles per discrepancy before closure

Risk-Based Quality Management (RBQM): • ICH E6(R2)/(R3) shifted monitoring away from 100% source data verification toward centralized statistical monitoring • Key Risk Indicators (KRIs) — query rate per site, AE reporting lag, protocol deviation rate — are monitored centrally; only outlier sites receive targeted on-site SDV • This reduces monitoring cost by an estimated 15–25% versus legacy 100%-SDV models while maintaining ALCOA+ data integrity standards

The 2022 TransCelerate RBQM benchmarking study found that trials using centralized statistical monitoring closed database lock 18% faster on average than trials relying on legacy 100% on-site SDV, with no measurable increase in post-lock data-quality findings during FDA inspection.

ALCOA+ and 21 CFR Part 11 as the governing data-integrity framework

Every cleaning action performed on trial data must remain defensible under regulatory inspection. The FDA and EMA anchor this expectation in two overlapping frameworks:

ALCOA+ (data integrity principles, originating from FDA guidance, expanded by WHO/MHRA): Attributable · Legible · Contemporaneous · Original · Accurate · plus Complete · Consistent · Enduring · Available

• Attributable: every edit in the EDC audit trail is tied to a named, authenticated user ID — no shared logins permitted • Contemporaneous: data entry timestamped at the moment of capture, not batch-entered days later • Original / Accurate: source-to-EDC transcription verified by SDV or, increasingly, direct data capture (eSource) that eliminates the transcription step entirely

21 CFR Part 11 (electronic records/signatures): • Full audit trail: every field change logged with old value, new value, user, timestamp, and reason-for-change • System validation: the EDC platform itself must be validated (IQ/OQ/PQ) before go-live • Electronic signatures: Investigator signs off on each CRF page cryptographically, binding accountability to the data

The consequence for SDTM downstream: nothing is deleted, only superseded. When a data manager corrects an implausible lab value, the SDTM SUPPQUAL or RELREC structure — or simply the audit-trailed EDC correction — preserves the original entry, so the full history remains reconstructable years later during an FDA Bioresearch Monitoring (BIMO) inspection.

MedDRA, WHO Drug Dictionary, and CDISC Controlled Terminology

Free-text verbatim terms — the messy, human-written strings clinicians and patients actually type — are the single largest source of downstream inconsistency in a clinical database. Medical coding translates this free text into hierarchical, standardized dictionaries, while CDISC Controlled Terminology constrains every categorical SDTM variable to a closed, regulator-recognized value set.

• 5: MedDRA hierarchy levels (SOC → HLGT → HLT → PT → LLT)
• v27.0: Current MedDRA version (updated biannually (Mar/Sep))
• ~500,000: WHO-DD Enhanced entries (trade + generic drug records)
• 80–90%: Auto-coding accuracy (algorithmic match to PT before manual review)

MedDRA coding — from verbatim adverse event text to Preferred Term

MedDRA (Medical Dictionary for Regulatory Activities), maintained by the ICH-sponsored MSSO, provides the standardized vocabulary that lets an FDA or EMA safety reviewer aggregate adverse events across every submitted trial in a therapeutic area:

Five-level hierarchy: • System Organ Class (SOC) — 27 top-level classes (e.g., "Cardiac disorders") • High Level Group Term (HLGT) • High Level Term (HLT) • Preferred Term (PT) — the primary reporting unit, ~26,000 terms • Lowest Level Term (LLT) — ~90,000 terms, closest to verbatim, maps 1:1 upward to a single PT

Coding workflow: 1. Verbatim term captured on AE CRF page exactly as investigator wrote it ("felt dizzy and lightheaded on standing") 2. Auto-coder (WebSDM, Oracle Central Coding, ARISg) proposes an LLT match via synonym/fuzzy matching against the MedDRA dictionary 3. Medical coder (often a nurse or pharmacist) reviews and confirms, or manually codes ambiguous/compound terms — a single verbatim describing two distinct events must be split into two AE records before coding 4. Primary SOC assignment: some PTs map to multiple SOCs; the coding convention designates one "Primary SOC" for the summary tables, with secondary SOC links retained in supplemental data

Dowstream SDTM impact: • AEDECOD (PT) and AEBODSYS (Primary SOC) become the core AE domain variables driving every safety table, figure, and listing (TFL) in the Clinical Study Report • MedDRA Standardised MedDRA Queries (SMQs) — curated groupings of related PTs (e.g., "Anaphylactic reaction" SMQ spans 60+ PTs across multiple SOCs) — power targeted safety signal searches that a naive PT-frequency table would miss entirely

WHO Drug Dictionary and CDISC Controlled Terminology codelists

Concomitant and prior medications follow a parallel coding pathway using the WHO Drug Dictionary Enhanced (WHO-DDE), maintained by the Uppsala Monitoring Centre (WHO-UMC):

WHO-DD structure: • Trade name → Anatomical Therapeutic Chemical (ATC) classification → Active ingredient • ATC provides a 5-level hierarchy analogous to MedDRA's SOC/PT structure, enabling cross-trial pooling of drug-exposure data by therapeutic class • A single trade name (e.g., regional brand of acetaminophen) maps to one standardized generic ingredient, essential when pooling multi-region, multi-language trials

CDISC Controlled Terminology (CT): • Published quarterly by CDISC in coordination with the NCI Enterprise Vocabulary Services (NCI EVS) • Covers non-drug, non-AE categorical variables across every SDTM domain: SEX (M/F/U/UNDIFFERENTIATED), RACE, LBTESTCD, VSTESTCD, route of administration, units of measure (extensible codelists follow UCUM where possible) • "Extensible" vs. "non-extensible" codelists: sponsors may add sponsor-specific terms to extensible lists (flagged in define.xml) but must use only the published values for non-extensible lists — using an unlisted value in a non-extensible codelist is an automatic Pinnacle 21 Error

Coding governance: • Coding conventions frozen in a Medical Coding Plan before database lock; MedDRA/WHO-DD version is locked for the entire study to avoid mid-study term-mapping drift • A full MedDRA/WHO-DD re-coding sweep is performed if the dictionary version becomes badly outdated between database build and lock — sponsors budget for at least one full re-coding pass per multi-year trial

A 2021 CDISC conformance survey of FDA submissions found that inconsistent MedDRA versions across pooled integrated-safety-summary studies were among the top five most common Study Data Technical Conformance Guide findings — reviewers explicitly flag mixed-dictionary-version pooling as a barrier to cross-study safety signal detection.

Mapping Cleaned, Coded Data into CDISC SDTM Domains

With queries closed and terminology coded, the transformation engine — typically SAS, R, or a Python/Pandas pipeline — reshapes source data into the ~24-domain SDTM structure defined by the Study Data Tabulation Model Implementation Guide (SDTM IG). This is where trial-specific CRF variables become the standardized, machine-readable structure every FDA reviewer expects to see, regardless of sponsor or therapeutic area.

• v3.4: SDTM IG current version (CDISC, published 2023)
• ~24: Core SDTM domains (DM, AE, LB, VS, EX, CM, MH, DS...)
• Req/Exp/Perm: Required variables per domain (three compliance tiers per SDTM IG)
• 150–400 pp: Typical mapping spec length (per study, per domain variable)

Domain classes and the general observation classes model

SDTM organizes all clinical data into three General Observation Classes, each with a fixed variable role structure that keeps every domain interoperable:

Interventions class (things done TO the subject): • EX (Exposure — actual study drug dosing), EC (Exposure as Collected), CM (Concomitant Medications), SU (Substance Use)

Events class (things that HAPPEN to the subject): • AE (Adverse Events), MH (Medical History), DS (Disposition), CE (Clinical Events)

Findings class (things MEASURED or OBSERVED about the subject): • LB (Laboratory), VS (Vital Signs), EG (ECG), QS (Questionnaires), PE (Physical Exam)

Every domain shares a common variable spine: • Identifier variables: STUDYID, DOMAIN, USUBJID (unique subject identifier, constructed as STUDYID-SITEID-SUBJID), --SEQ (sequence number ensuring row uniqueness) • Topic variable: the "what" — AETERM, CMTRT, LBTESTCD • Qualifier variables: --CAT/--SCAT (category/subcategory), --SEV or --TOXGR (severity/toxicity grade), --STRESC/--STRESN (standardized result, character/numeric) • Timing variables: --DTC (ISO 8601 date/time), --DY (study day, signed integer relative to RFSTDTC reference start date), VISITNUM/VISIT

Derivation logic — study day (--DY): • --DY = (date portion of --DTC) − (date portion of RFSTDTC) + 1, if --DTC ≥ RFSTDTC • --DY = (date portion of --DTC) − (date portion of RFSTDTC), if --DTC < RFSTDTC (no day-zero, so pre-treatment days are negative with no zero crossing) • This single derivation rule, applied identically across every Findings/Events domain, is what lets a reviewer instantly locate any event relative to first dose without cross-referencing a separate calendar

Required/Expected/Permissible (Req/Exp/Perm): • Required variables (e.g., USUBJID, --SEQ) must be populated in every record — a missing Required variable is an automatic Pinnacle 21 Reject • Expected variables should be populated when the source data supports it; a systematic gap triggers a Warning • Permissible variables are included only when relevant (e.g., AEACN — action taken with study treatment — only applies to AE records where a dosing action occurred)

The Trial Design Model and cross-domain traceability (RELREC)

Beyond subject-level data, SDTM requires a parallel Trial Design Model (TDM) describing the protocol itself as structured metadata:

Trial Design domains: • TA (Trial Arms) — defines each treatment arm as an ordered sequence of Elements • TE (Trial Elements) — defines each distinct time period (Screening, Treatment, Follow-up) • TV (Trial Visits) — the planned visit structure and visit windows • TS (Trial Summary) — study-level parameters (e.g., planned enrollment, therapeutic area, blinding schema) coded against CDISC-controlled TSPARMCD codelist

Cross-domain traceability — RELREC: • Clinical events rarely live in a single domain. An adverse event that led to a dose interruption connects an AE record to an EX record; RELREC (Related Records) explicitly links them via shared identifier keys (USUBJID + --SEQ pairs) • SUPPQUAL (Supplemental Qualifiers): non-standard variables that don't fit the parent domain's fixed structure are appended via SUPPQUAL rather than inventing new domain columns — keeps the core domains conformant while preserving sponsor-specific data

Derived vs. collected data separation: • SDTM is explicitly a tabulation model, not an analysis model — it should stay close to what was actually collected or minimally derived (unit conversions, coding, standard date formatting) • Complex statistical derivations (change from baseline, treatment-emergent flags, population flags) belong in ADaM (Analysis Data Model) datasets built downstream from SDTM, not in SDTM itself • This separation of concerns is why regulatory reviewers can independently reconstruct any ADaM analysis result by tracing back through ADaM define.xml → SDTM define.xml → aCRF, without ever touching the sponsor's original SAS analysis code

FDA's Study Data Technical Conformance Guide explicitly states that mixing derived/analysis-only variables into SDTM domains is one of the most frequent sources of Refuse-to-File technical rejections at the pre-submission validation stage — reviewers expect SDTM to remain a faithful, minimally-transformed tabulation of what was collected.

Pinnacle 21 Validation and Define.xml Metadata Traceability

No SDTM package reaches a regulator without first passing automated conformance validation. Pinnacle 21 (Community edition, free; Enterprise, licensed) is the de facto industry-standard validator, running the complete dataset bundle against the SDTM IG structural rules, CDISC Controlled Terminology, and the FDA/PMDA Validator Rules that mirror what the agency's own JReview and Empirica-based reviewer tools will check on receipt.

• Reject/Error/Warning/Notice: P21 rule categories (four severity tiers)
• 40–120: Typical first-pass findings (per study across all domains)
• 0 Rejects, 0 Errors: Target before submission (Warnings justified in SDRG)
• v2.1: Define.xml current version (CDISC ODM-based metadata standard)

Pinnacle 21 rule severities and the remediation workflow

Pinnacle 21 executes several thousand individual conformance checks encoded from the SDTM IG, the CDISC CT release, and the FDA/PMDA Validator Rules published alongside the Study Data Technical Conformance Guide:

Severity tiers: • Reject — structural violations that will cause the FDA's own gateway validator to bounce the submission outright (e.g., missing required domain, invalid XPT structure, duplicate --SEQ values within a subject) • Error — SDTM IG non-conformance that a reviewer will flag and may request be fixed before substantive review begins (e.g., non-extensible codelist value used, --DTC not ISO 8601, orphaned RELREC key) • Warning — deviations that are permitted but must be explained (e.g., an Expected variable largely null, an unusually long free-text qualifier) • Notice — informational, no action required, but visible to the reviewer

Remediation loop: 1. Run P21 against the full dataset bundle → generates an Excel/HTML issue summary, one row per finding with dataset, variable, and record-level detail 2. Data programmers triage: true SDTM-mapping bugs get fixed at the transformation-spec level (not patched dataset-by-dataset, to avoid mapping drift between re-runs) 3. Legitimate Warnings that cannot or should not be "fixed" (e.g., a genuinely sparse Expected variable because the protocol didn't collect it for a subgroup) are documented with rationale in the SDTM Reviewer's Guide (SDRG) 4. Full re-run after every SDTM programming change — validation is iterative, typically 4–8 full cycles between first draft and submission-ready

A typical Phase III oncology study starts its first P21 run with 60–120 findings across Reject/Error/Warning and is driven to zero Rejects and zero unexplained Errors before the biostatistics/regulatory affairs sign-off gate.

Define.xml — the machine-readable map from dataset to origin

Define.xml (built on the CDISC Operational Data Model, current version 2.1) is the single metadata file that lets an FDA reviewer's tooling — or a human reviewer in a PDF viewer — trace every variable in every SDTM dataset back to where it came from:

What define.xml encodes: • Full dataset-level metadata: domain, structure, class, purpose (Tabulation), key variables • Full variable-level metadata: name, label, data type, controlled terminology codelist reference, origin (CRF page reference, derived, assigned, or protocol-defined), and — critically — the derivation algorithm in plain text for any computed variable • Value-level metadata: for variables like --TESTCD/--TEST that take many values (e.g., every distinct lab test), define.xml specifies per-value constraints (e.g., LBSTRESN units differ by LBTESTCD) • Hyperlinked traceability: define.xml is rendered as an interactive HTML page (via the CDISC define.xml stylesheet) where clicking a variable jumps to the exact annotated CRF (aCRF) page image showing where that data point was captured

The annotated CRF (aCRF): • A PDF of the blank CRF with every field hand-annotated (typically in a distinct color) showing the exact SDTM domain.variable it maps to • Cross-referenced bidirectionally with define.xml — a reviewer questioning a specific AE record can open the aCRF, find the original field, and understand precisely how AEDECOD was derived from the verbatim entry

Together, define.xml + aCRF + SDRG let a reviewer independently audit the entire cleaning-to-submission pipeline without ever asking the sponsor a clarifying question — which is precisely the point.

FDA's Office of Study Integrity and Surveillance has stated in public CDISC forums that incomplete or inconsistent define.xml origin metadata is among the most common reasons SDTM packages require a formal Information Request during standard review — directly extending review-cycle time for the sponsor.

Assembling and Transmitting the Final Submission Package

The last stage converts a validated SDTM dataset bundle into the exact electronic package format a regulator's intake system expects — SAS XPT v5 transport files, structured metadata, and narrative reviewer guides — packaged into the eCTD (electronic Common Technical Document) and transmitted through the FDA Electronic Submissions Gateway (ESG) or the EMA's equivalent eSubmission Gateway.

• XPT v5: Transport file format (SAS Transport, 8-char variable name limit)
• 5.3.5.1: eCTD Module for study data (tabulation datasets + define.xml + aCRF)
• ~2–5 days: FDA technical rejection window (automated gateway validation)
• since 2017: Study Data Standards mandate (FDA binding guidance for NDA/BLA)

Package assembly — XPT transport files and the Reviewer's Guide

Final packaging converts validated, in-memory SDTM datasets into the literal file formats the regulator's systems ingest:

SAS Transport (XPT) v5 constraints: • Variable names limited to 8 characters (a legacy constraint inherited from SAS Transport format v5, still mandated by FDA despite v3.4 SDTM IG supporting longer names in principle) • Each dataset saved as its own .xpt file, one file per SDTM domain, named to match the two-letter domain code (dm.xpt, ae.xpt, lb.xpt...) • File size and record-count practicalities: extremely large Findings domains (LB, VS, EG for a large longitudinal trial) may require splitting into multiple XPT files per domain, tracked in define.xml

The SDTM Reviewer's Guide (SDRG): • A structured narrative document (CDISC-published template) that walks the FDA reviewer through study-specific mapping decisions, known limitations, unresolved Pinnacle 21 Warnings with justification, and any non-standard domain or naming convention used • Functions as the human-readable companion to the machine-readable define.xml — reviewers are instructed to read the SDRG first before diving into the datasets

Assembly checklist before transmission: • All domain XPT files + define.xml + define.pdf (bookmarked, hyperlinked rendering) + aCRF PDF + SDRG + ADaM equivalent bundle (ADaM datasets, ADaM define.xml, ADRG) + Analysis Results Metadata (ARM) for key efficacy/safety tables • Final full Pinnacle 21 run against the exact files being transmitted — not an earlier working copy — is standard practice to avoid last-minute drift bugs

eCTD structure, gateway transmission, and technical rejection risk

The eCTD is the internationally harmonized (ICH) electronic dossier format that structures an entire marketing application — SDTM/ADaM data is only one branch of a much larger tree:

eCTD module structure: • Module 1: region-specific administrative information • Module 2: summaries (quality, nonclinical, clinical overview) • Module 3–4: quality (CMC) and nonclinical study reports • Module 5: clinical study reports — Section 5.3.5.1 holds the tabulation datasets (SDTM), 5.3.5.3 the analysis datasets (ADaM)

Transmission and automated technical validation: • Sponsors submit via the FDA Electronic Submissions Gateway (ESG) using AS2 protocol, or the equivalent EMA eSubmission Gateway for EU procedures • On receipt, FDA runs the Technical Rejection Criteria (TRC) validation — an automated check for file integrity, virus scanning, valid eCTD backbone XML, and baseline SDTM/define.xml structural presence • A Technical Rejection at this stage (distinct from a scientific Refuse-to-File) typically returns within 2–5 business days and requires full package resubmission — timelines that sponsors build explicit buffer around ahead of PDUFA-driven filing deadlines

Post-submission reviewer tooling: • FDA reviewers load the SDTM/ADaM package into internal tools (e.g., JReview, Certara's tools, or custom R/SAS pipelines) that consume the datasets directly using define.xml metadata to auto-generate variable dictionaries • A clean, fully conformant package with zero unexplained Pinnacle 21 findings materially shortens the reviewer's data-reconciliation effort — the entire point of standing up this pipeline in the first place is to make the regulator's job as close to friction-free as the underlying science allows

FDA's Study Data Standards guidance (binding since December 2016 for NDAs/BLAs and 2017 for INDs) made CDISC SDTM/ADaM submission functionally mandatory for nearly all US marketing applications — sponsors who submit non-conformant legacy data formats now face an explicit Refuse-to-File risk rather than simply a reviewer inconvenience.