Clinical Risk Model Calibration Simulator

Model Miscalibration

Confidence gain 1.00×

Systematic offset 0.00

Cohort

Patients 1200

Bins 10

Live Metrics

Brier score

–

ECE

–

AUC

–

Legend

Predicted-probability bar (model) Perfect-calibration diagonal Patient — event occurred Patient — no event

How it works

Each simulated patient has a true, unobservable risk p_true = σ(1.1·x − 0.4) from a hidden risk factor x, and their real outcome is a Bernoulli draw from that risk. A clinical-decision-support model never sees p_true — it outputs its own predicted probability:

p_model = σ( gain · logit(p_true) + offset )
σ(z) = 1 / (1 + e^-z)

Confidence gain > 1 makes the model overconfident (predictions pushed toward 0 or 1); gain < 1 makes it underconfident (predictions squeezed toward 0.5). Systematic offset shifts every prediction up or down regardless of the true risk — exactly the failure mode an AI risk score shows when it is deployed on a population that differs from the one it was trained on.

Patients are grouped into equal-width probability bins. For each bin the bar height is the observed event rate; its horizontal position is the model's average predicted probability. A perfectly calibrated model has every bar top touching the diagonal reference line.

Brier score = (1/N) Σ (p_model,i − outcome_i)²
ECE = Σ_bins (n_bin/N) · |observed_bin − predicted_bin|
AUC = (rank-sum of positives − nPos(nPos+1)/2) / (nPos · nNeg)

Brier score — overall accuracy of the probabilities (lower is better; 0 is perfect).
ECE — average gap between confidence and reality (calibration only; 0 is perfect).
AUC — the model's ability to rank sick patients above healthy ones (discrimination), unaffected by gain/offset since those are monotonic transforms.

This is the exact tension real hospital deployments hit: an AI sepsis or readmission-risk score can have excellent AUC yet still be dangerously miscalibrated, triggering far too many — or too few — clinical alerts at any fixed threshold.

← predicted probability → ↑ observed / predicted rate

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.