Just-in-time adaptive intervention: contextual bandits and receptivity modeling pick the optimal moment to deliver a mobile-health nudge
Just-in-Time Adaptive Interventions (JITAIs) — a framework formalized by Nahum-Shani, Smith, Spring, Collins, Witkiewitz, Tewari & Murphy (Annals of Behavioral Medicine, 2018) — depend on knowing, at every moment, what state the user is actually in. Before any model or algorithm runs, a continuous stream of passive smartphone sensor data must be compressed into a compact, structured context vector describing location, activity, device state, and calendar availability.
The JITAI framework introduces a precise vocabulary that this entire pipeline is built around:
• Decision points: moments in time at which the system decides whether and how to intervene (e.g., every 30 minutes, or whenever a new context is detected) • Tailoring variables: the context features used to decide whether/what to deliver — location, activity, stress, time since last message • Intervention options: the discrete set of things that could be delivered (a step-count nudge, a medication reminder, nothing at all) • Decision rules: the function mapping tailoring variables to an intervention option — this is exactly the receptivity model plus bandit policy built in stages 2 and 3
This vocabulary matters because it separates "when to look" (decision points) from "what to decide" (decision rules) — a distinction that keeps the system computationally tractable on-device.
A typical mobile-health context pipeline fuses several independent sensor streams:
• GPS / significant-location clustering: raw lat/long is clustered (DBSCAN-style) into semantic places — home, work, gym — without ever storing a street address, preserving privacy while enabling "at work" as a feature • Accelerometer + gyroscope activity recognition: a lightweight on-device classifier (walking, running, still, in-vehicle) runs at 20–50 Hz and is down-sampled to a rolling 1-minute activity label • Screen and app-foreground events: screen-on/off timestamps and foreground app category (messaging, social, none) proxy for interruptibility • Calendar and day-of-week: busy/free blocks and circadian phase (morning/midday/evening/night) anchor the context in the user's schedule rather than raw clock time alone
Each of these streams is noisy and intermittent on its own — GPS drops indoors, accelerometer misclassifies stationary-in-car as still — so no single signal is trusted; they are fused downstream.
Raw sensor events are aggregated into fixed windows (typically 5–15 minutes) and turned into a numeric feature vector: proportion of the window spent stationary, distance from home centroid, minutes since last unlock, hour-of-day encoded cyclically (sin/cos), day-of-week one-hot, calendar busy flag.
Two engineering constraints dominate real deployments: battery budget (continuous GPS at high frequency drains the device in hours, so sampling is adaptively throttled) and privacy (in trials like HeartSteps and Sense2Stop, feature extraction runs on-device and only derived, de-identified features — never raw GPS traces — are transmitted to the server). This context vector is the sole input to the receptivity model built in Stage 2.
Knowing the context is not enough — the engine needs to know how likely the user is to actually engage with a message sent right now. Receptivity modeling, studied extensively in mobile-sensing research (Pejovic & Musolesi's InterruptMe, and receptivity analyses layered onto the HeartSteps and Sense2Stop trials), fits a classifier that turns the context vector into a single probability of engagement versus ignoring or dismissing.
The core model estimates P(engage=1 | x) where x is the Stage-1 context vector. Two model families dominate mobile-health deployments:
• Logistic regression: P = 1 / (1 + e^-(w·x + b)) — interpretable, cheap to run on-device, coefficients directly readable as "driving reduces receptivity by w_drive" • Gradient-boosted trees (XGBoost / LightGBM): capture non-linear interactions (e.g., "stationary AND evening AND weekday" behaves differently than any single feature) at the cost of interpretability
A persistent challenge is class imbalance: across most JITAI deployments roughly 80–90% of delivered notifications are ignored or dismissed, so naive accuracy is a poor metric. Evaluation instead relies on AUC-ROC (area under the receiver-operating curve) and calibration curves, since the raw probability — not just the rank order — is consumed directly as a reward signal by the bandit in Stage 3.
Across published mobile-sensing studies, a consistent set of features carries most of the predictive signal:
• Activity state: driving or in-transit sharply suppresses receptivity (interacting with a phone while driving is both dangerous and rare); stationary-at-home or stationary-at-work raises it • Time since last phone unlock: very recent unlock (<2 min) means the phone is already in-hand — the single strongest predictor in several published receptivity models • Recent engagement history: a user who dismissed the last two messages is markedly less likely to engage with a third within the hour — the seed of the habituation-decay logic formalized in Stage 4 • Time-of-day and day-of-week: receptivity typically shows two to three daily peaks (post-wake, midday, evening) separated by troughs during commute and focused-work blocks • Weather and season: secondary but measurable effects on outdoor-activity nudges specifically
Because the raw model score is fed forward as a reward prior into a bandit that makes real delivery decisions, calibration matters more than raw discrimination. Platt scaling or isotonic regression is typically applied post-hoc so that "P(receptive)=0.6" really does correspond to roughly 60% observed engagement in that bucket, not just a higher-than-average score.
In clinical and behavioral-health deployments, interpretability tools (SHAP feature attributions, partial-dependence plots) are also produced alongside the score so that study clinicians can audit why the model favors certain moments — a requirement that repeatedly surfaces in IRB-reviewed mobile-health trials such as Sense2Stop, where opaque black-box timing decisions are difficult to justify to a human review board.
A receptivity score alone only tells you whether now is a good moment — it does not say what to send. The engine frames each candidate intervention type as an "arm" of a contextual multi-armed bandit and must balance exploiting the arm currently believed best against exploring arms whose true value is still uncertain — the same exploration/exploitation trade-off first formalized by Thompson (1933) and adapted to personalized recommendation by Li, Chu, Langford & Schapire's LinUCB (WWW, 2010).
At each decision point the engine must choose one of K discrete intervention options — for example: an activity nudge, a medication reminder, an ecological-momentary-assessment (EMA) survey prompt, or a social-connection cue — or the option to send nothing at all. Choosing an arm yields a stochastic reward (1 if the user engages, 0 if they dismiss or ignore), and the goal over many decision points is to minimize cumulative regret: the gap between the reward actually collected and the reward an oracle who always knew the best arm would have collected.
The "contextual" part is essential: the best arm is not fixed — a medication reminder is high-value at a scheduled dosing time and low-value at 2am, so the bandit conditions its choice on the Stage-1/2 context vector rather than treating all decision points identically.
Thompson sampling maintains a Bayesian posterior over each arm's success probability, typically a Beta distribution Beta(α_a, β_a) for arm a, initialized at α=β=1 (a uniform prior — total uncertainty).
At each decision point: 1. Draw one sample θ_a ~ Beta(α_a, β_a) for every arm a 2. Select the arm with the highest sampled θ_a 3. Observe the reward r ∈ {0,1} after delivery 4. Update: α_a ← α_a + r, β_a ← β_a + (1−r)
Early on, posteriors are wide (few observations) so sampled θ values vary a lot and the policy naturally explores. As evidence accumulates, posteriors narrow around the true success rate and the policy naturally converges to exploitation — no explicit ε-greedy schedule is required, though an exploration-rate parameter can still widen the effective sampling variance to keep exploring in a non-stationary environment where user routines drift.
LinUCB assumes the expected reward of arm a is linear in the context vector x: E[r | x, a] = θ_a · x, and maintains a ridge-regression estimate θ̂_a from past (context, reward) pairs for that arm. At each decision point it selects the arm maximizing an upper-confidence-bound score:
UCB_a(x) = θ̂_a·x + α·√(xᵀ A_a⁻¹ x)
where A_a is the running design matrix for arm a and α controls how aggressively uncertainty is rewarded (higher α ⇒ more exploration of under-tried context regions). Originally built for personalized news-article recommendation at Yahoo (Li et al. 2010), LinUCB transfers naturally to nudge timing: instead of "which article for this reader," it answers "which intervention for this context," and its confidence term automatically explores harder in contexts — like a newly-observed commute pattern — where the model has little data.
A subtlety of online bandit learning in health interventions is that naive greedy policies can produce biased effect estimates — if the algorithm only ever sends reminders when it already predicts success, it becomes impossible to later prove the reminder caused the outcome. Klasnja, Hekler, Shiffman, Boruvka, Almirall, Tewari & Murphy's micro-randomized trial (MRT) design (2015) addresses this directly: at each decision point, the intervention is randomized with a known, context-dependent probability (commonly in the 0.3–0.7 range) rather than deterministically optimized.
This controlled randomization — layered underneath the bandit's preference — is what let the HeartSteps trials produce statistically valid, causal estimates of each suggestion type's "proximal effect" on the following 30 minutes of step count, rather than merely correlational ones.
An unconstrained bandit, left to purely maximize predicted reward, will over-deliver: it will keep pinging the arm that has worked best so far until the user is exhausted by it. Real deployments wrap the learned policy in hard, non-negotiable guardrails — grounded in notification-fatigue research such as Pielot, Church & de Oliveira's "Didn't you see my message?" (MobileHCI, 2014) and Mehrotra et al.'s large-scale notification studies — that cap volume regardless of what the algorithm believes is optimal.
Notification fatigue is well documented: Pielot et al. found that as daily notification volume rises, both the fraction attended to and the reported value users place on each individual notification fall — a habituation effect familiar from any repeated-stimulus psychology. In a JITAI context this is doubly costly: a habituated user not only ignores the current message, they become less receptive to future ones, actively degrading the very receptivity distribution the Stage-2 model is trying to learn.
Because the bandit in Stage 3 only ever observes reward = engaged/ignored for messages it actually sends, an unconstrained policy that discovers "sending constantly" temporarily looks fine in the metrics right up until cumulative fatigue collapses engagement — a delayed, non-stationary cost that a myopic bandit cannot see coming on its own.
Production JITAI systems separate "what the algorithm wants" from "what is allowed to actually happen" via a guardrail layer applied after arm selection:
• Quiet hours: no delivery during a fixed overnight window (commonly 22:00–07:00) regardless of predicted receptivity — sleep-time interruption is both intrusive and clinically counter-indicated for many behavioral targets • Daily notification cap: a hard ceiling (e.g., 3–5 per day) enforced independent of how many high-value opportunities the bandit believes exist • Minimum cooldown window: a fixed number of minutes must elapse between any two deliveries, preventing bursts even inside a single high-receptivity block • Novelty rotation: arms that were delivered most recently are down-weighted for the next few decision points to counter within-arm habituation specifically • Immediate opt-out honoring: a user snooze/mute signal suppresses delivery instantly, overriding every other rule
These constraints are intentionally simple and auditable — unlike the bandit's learned weights, they are hand-set thresholds that a clinician or IRB can inspect and sign off on directly.
To make fatigue visible to the learning system rather than only enforced externally, the effective reward used to update the bandit posterior is discounted by a habituation term:
r_eff = r · exp(−λ · exposures_since_last_engagement)
where exposures_since_last_engagement counts consecutive ignored deliveries of that arm and λ is a decay rate fit from historical dismissal patterns. A single engagement resets the counter to zero. This lets the posterior itself reflect "this arm is currently fatigued" without needing a separate rule, and it naturally pushes Thompson sampling toward a different, fresher arm the next time a decision point arrives — guardrails and learning working in the same direction rather than fighting each other.
The final stage closes the loop: whether the user engaged with or dismissed each delivered nudge is logged as a reward, and both the bandit posterior and the underlying receptivity model are updated in real time. Tomorrow's decision points inherit today's evidence — this continual online-learning cycle is exactly the closed-loop architecture used in the HeartSteps V2/V3 trials led by Klasnja, Nahum-Shani, and Murphy.
The simplest reward is binary: did the user open/act on the notification within a fixed window (e.g., 5–30 minutes)? Richer deployments shape the reward further — dwell time on the app after opening, whether an EMA survey was actually completed rather than merely opened, or the downstream behavioral outcome itself (step count in the following 30 minutes, as measured directly in HeartSteps).
Delayed and partial reward attribution is a genuine challenge: if a user opens a step-count nudge but only starts walking twenty minutes later, naive short-window reward definitions can undercount true effect, biasing the bandit against arms with slower-acting but real downstream impact.
Each logged outcome triggers an incremental Bayesian update — for Thompson sampling this is the α/β increment described in Stage 3, applied within minutes of the outcome being observed, not batched overnight. This is what makes the system "online" rather than a periodically-retrained batch model.
A persistent complication is non-stationarity: a user's routine drifts over weeks (a new job, a change of season, travel), so evidence gathered a month ago can actively mislead a stationary posterior. Production systems address this with either a sliding window (only the last N days count) or explicit discounting (older observations are down-weighted exponentially), trading faster adaptation against noisier short-run estimates — a bias-variance trade-off with no universal answer.
Across published JITAI deployments, algorithmically-timed nudges consistently outperform fixed, un-personalized schedules — commonly cited relative engagement lifts fall in the 20–40% range, alongside measurable behavioral effects such as HeartSteps' step-count increase in the window following a well-timed suggestion. Critically, the guardrail layer from Stage 4 is what keeps these gains from being erased by fatigue over a multi-week or multi-month deployment, which is the timescale that matters for chronic-condition management (diabetes, hypertension) rather than a single novel-app-effect week.
The same closed-loop architecture — context sensing, receptivity modeling, contextual bandit selection, guardrails, and online feedback — now underlies a broad family of digital-health products beyond activity coaching: medication-adherence reminders, mental-health check-in prompts, and smoking-cessation lapse interventions (as studied in Sense2Stop) all instantiate the same five stages with different arms and different reward definitions.
The HeartSteps trials demonstrated that this exact five-stage loop — context sensing, a calibrated receptivity model, contextual-bandit arm selection, hard fatigue guardrails, and continuous online feedback — can sustain step-count increases across a multi-week deployment without the engagement collapse that plagues fixed-schedule notification strategies. The guardrail layer is not a minor implementation detail: published analyses consistently show that removing daily caps and cooldowns erodes long-run engagement even when the bandit itself is learning correctly, because the reward signal the bandit optimizes cannot see fatigue it was never allowed to cause.