Behavioral signal modeling to flag DTx users at risk of dropout before they lapse
Unlike a pharmacy refill record, which gives a single binary adherence signal per prescription cycle, digital therapeutics generate rich, continuous behavioral telemetry — every app open, every completed module, every skipped scheduled activity is a timestamped event. This granularity is what makes real-time dropout prediction possible in DTx, in contrast to traditional medication adherence monitoring which is typically retrospective.
Production DTx adherence-monitoring pipelines typically ingest several categories of raw event data:
• Session-level events: app open/close timestamps, session duration, screen-by-screen navigation path • Module completion events: which therapeutic content (CBT lesson, exposure exercise, breathing session) was started, completed, or abandoned mid-way, with timestamp and duration • Scheduled activity events: whether a prompted check-in, symptom log, or homework assignment was completed on time, late, or missed entirely • Passive sensor data (where available): wearable-derived sleep, activity, or heart-rate variability signals that can proxy for symptom state independent of app engagement itself • Notification interaction: whether push notifications were opened, dismissed, or ignored — itself a leading indicator distinct from core app usage
The key design principle is that raw events alone are not predictive — a single missed session means little. It is the pattern across many events, aggregated over rolling windows, that carries predictive signal, which is why the next stage (feature engineering) is where most of the modeling value is actually created.
The difference between a mediocre and a strong dropout-prediction model is rarely the choice of algorithm — it is almost always the quality of engineered features. Rolling-window behavioral trends consistently outperform static, single-point-in-time features in published digital-health churn literature.
Most production DTx dropout models converge on a similar core feature set, echoing patterns well established in subscription-business churn modeling but adapted for clinical context:
• Session-frequency slope: the trend (not the absolute level) of sessions-per-week over a rolling 7- and 14-day window. A declining slope is a substantially stronger predictor than a low absolute session count, because it captures the trajectory rather than a single snapshot — a user with 2 sessions/week trending down from 5 is higher risk than a stable 2/week user • Days-since-last-open (recency): a classic RFM (recency-frequency-monetary, adapted here as recency-frequency-completion) feature; for a program designed around daily or near-daily engagement, 3+ consecutive days of inactivity is a strong dropout precursor • Module completion lag: ratio of actual progress through the therapeutic curriculum versus the expected pace given elapsed enrollment time; falling more than ~30–40% behind expected pace correlates strongly with eventual dropout in published DTx engagement analyses • Check-in miss streak: consecutive missed scheduled check-ins (symptom logs, homework); this feature captures a specific behavioral pattern — early sporadic misses are common and often benign, but 2+ consecutive misses compound risk multiplicatively rather than additively • Time-of-day/day-of-week regularity: entropy of session timing; users whose engagement times become erratic (versus a previously consistent routine) show elevated dropout risk even before frequency itself declines — often the earliest available signal
Feature engineering windows must balance responsiveness (short windows catch changes fast but are noisy) against stability (long windows are stable but lag real behavior change) — most production systems use an ensemble of multiple window lengths rather than picking one.
Two modeling paradigms dominate real-world DTx dropout prediction: discrete-time survival analysis, which directly models time-to-event and handles censoring correctly, and gradient-boosted binary classifiers, which are simpler to implement and tune but require careful handling of the time-varying nature of risk.
A naive binary classifier ("will this user drop out in the next 7 days: yes/no") trained on historical data has a subtle but important flaw: at the time of training, many currently-active users have not dropped out yet, but that does not mean they never will — they are "right-censored" observations. Treating them as clean negative examples biases the model toward underestimating risk for users who simply haven't reached their dropout point yet at data-collection time.
Discrete-time survival analysis (a person-period logistic regression formulation, or a more flexible framework like a discrete-time hazard model) explicitly models the hazard — the probability of dropout in period t given survival to period t — handling censored (still-active) users correctly by contributing only to the periods they were actually observed, without falsely labeling them as negative for future periods not yet observed.
In practice, many production teams use a pragmatic hybrid: a gradient-boosted tree classifier (XGBoost, LightGBM) trained to predict "dropout within next 7-day window" using a sliding-window resampling scheme that approximates the survival framework's handling of censoring, gaining the survival model's statistical correctness benefits while retaining the ease of feature engineering, hyperparameter tuning, and explainability tooling (SHAP values) that come with standard classifiers. Reported AUCs in published DTx and broader digital-health churn literature typically fall in the 0.72–0.85 range — meaningfully better than chance, but with enough residual error that threshold-setting (Stage 4) remains a genuine operational tradeoff rather than a solved problem.
A trained model outputs a continuous risk score; converting that into an actionable "flag this user" decision requires choosing a threshold, and that choice is fundamentally an operational and clinical tradeoff, not a purely statistical one — care-team capacity is finite, and every flagged user consumes real staff time.
Moving the classification threshold lower catches more true positive dropout risks (higher recall/sensitivity) but also flags more users who would have stayed engaged anyway (lower precision), generating unnecessary outreach and consuming scarce care-team time on users who did not actually need it. This is not merely a statistical nuisance — care teams in digital health programs are typically staffed to handle a fixed weekly outreach volume, and flooding them with low-precision flags produces "alert fatigue," a well-documented phenomenon in clinical decision-support literature where overwhelmed staff begin deprioritizing or ignoring flags altogether, degrading the entire system's effectiveness regardless of the underlying model's statistical quality.
Most production systems therefore set thresholds not purely by maximizing a statistical metric like F1-score, but by explicitly constraining the flagged population to a size the care team can realistically act on within its response-time SLA — commonly targeting roughly 10–20% of the active cohort flagged per week — then optimizing the model's precision within that fixed-capacity envelope. Some systems add a second-tier prioritization layer on top of the binary flag: ranking flagged users by a combination of risk score and estimated intervention value (e.g., deprioritizing users very close to program completion, where a missed final week matters less clinically than an early-program lapse), so care-team effort concentrates where it plausibly changes outcomes most.
A dropout prediction is only valuable if it triggers an intervention that actually changes the outcome. The final stage of the pipeline — and the one most often under-instrumented — is measuring whether flagged, contacted users are meaningfully less likely to drop out than similarly-flagged users who, for capacity or randomization reasons, were not contacted.
A common analytical trap is to compare outcomes for "contacted" versus "not contacted" flagged users and attribute the entire outcome difference to the outreach itself — but users who happen to get contacted faster, or who are more responsive to outreach, may differ systematically from those who are not, confounding any naive comparison. Rigorous DTx operations teams address this by maintaining a small randomized holdout group of flagged users who are deliberately not contacted (or contacted with a delay), allowing a true causal estimate of the outreach intervention's effect via a standard treatment-versus-control comparison, echoing randomized encouragement-design methodology from behavioral economics and public health outreach literature.
Published results from care-team outreach programs (spanning both DTx-specific and broader chronic-disease remote-monitoring contexts) generally report a 20–35% relative reduction in near-term dropout among proactively contacted at-risk users compared to matched or randomized controls, though effect sizes vary considerably by contact modality (a live phone call from a clinician-level coach typically outperforms an automated SMS nudge) and by how quickly contact is initiated after flagging — outreach initiated within 24–48 hours of the risk flag consistently outperforms delayed contact in the limited controlled evidence available, reinforcing why response-time SLAs are treated as a first-class operational metric alongside the underlying model's AUC.
A well-calibrated dropout model is necessary but not sufficient — published evaluations consistently find that intervention design and speed-to-contact explain as much variance in re-engagement success as model accuracy itself, meaning DTx operations teams that invest only in model refinement while neglecting outreach logistics typically see disappointing real-world impact despite strong offline model metrics.