Each simulated patient has a true, unobservable risk ptrue = ฯ(1.1ยทx โ 0.4) from a hidden risk factor x, and their real outcome is a Bernoulli draw from that risk. A clinical-decision-support model never sees ptrue โ it outputs its own predicted probability:
p_model = ฯ( gain ยท logit(p_true) + offset )
ฯ(z) = 1 / (1 + e^-z)
Confidence gain > 1 makes the model overconfident (predictions pushed toward 0 or 1); gain < 1 makes it underconfident (predictions squeezed toward 0.5). Systematic offset shifts every prediction up or down regardless of the true risk โ exactly the failure mode an AI risk score shows when it is deployed on a population that differs from the one it was trained on.
Patients are grouped into equal-width probability bins for the reliability diagram (top-left panel). For each bin the bar height is the observed event rate; its horizontal position is the model's average predicted probability. A perfectly calibrated model has every bar top touching the diagonal reference line.
Brier score = (1/N) ฮฃ (p_model,i โ outcome_i)ยฒ
ECE = ฮฃ_bins (n_bin/N) ยท |observed_bin โ predicted_bin|
AUC = (rank-sum of positives โ nPos(nPos+1)/2) / (nPos ยท nNeg)
The ROC panel (top-right) sweeps every possible alert threshold and plots the false-positive rate against the true-positive rate; the area under that curve is mathematically the same rank-sum AUC above (a standard equivalence โ gain/offset are monotonic transforms of the score, so they reshape the reliability diagram without moving the ROC curve or AUC at all, since ranking is unaffected).
The alert strip (bottom) plots every patient at their predicted probability; the threshold line marks where the model would fire a clinical alert. Sliding it trades sensitivity (catching real events) against specificity (not crying wolf on healthy patients) โ the exact dial a hospital sets when it deploys a sepsis or readmission-risk alert.
- Brier score โ overall accuracy of the probabilities (lower is better; 0 is perfect).
- ECE โ average gap between confidence and reality (calibration only; 0 is perfect).
- AUC โ the model's ability to rank sick patients above healthy ones (discrimination), unaffected by gain/offset.
- Sensitivity / Specificity / PPV โ depend on both discrimination and the chosen alert threshold, and shift when gain/offset are miscalibrated even though AUC does not.
This is the exact tension real hospital deployments hit: an AI sepsis or readmission-risk score can have excellent AUC yet still be dangerously miscalibrated, triggering far too many โ or too few โ clinical alerts at any fixed threshold.