500 synthetic samples each get a true latent probability q ~ Uniform(0,1) and a label y ~ Bernoulli(q). A classifier's raw predicted probability is a distorted version of q:
p_raw = σ( k · logit(q) + noise )
logit(q) = ln( q / (1−q) )
k = 1 would be a perfectly calibrated classifier. k > 1 makes it overconfident (pushes probabilities toward 0 or 1 faster than the data justifies) — the classic failure mode of deep nets and boosted trees. k < 1 makes it underconfident (predictions huddle near 0.5).
Reliability diagram: predictions are sorted into B equal-width bins by predicted probability. For each bin b:
confidence_b = mean predicted p in bin b
accuracy_b = fraction of true positives in bin b
This 2D plot draws the reliability curve directly on x = confidence, y = accuracy axes — a perfectly calibrated model traces the diagonal. Below it, a jittered strip plot shows every one of the 500 individual samples positioned by predicted probability and colored by its true label, so you can see the raw data the curve is summarizing.
Brier score = (1/N) Σ (p_i − y_i)² (lower is better)
ECE = Σ (n_b/N) · |accuracy_b − confidence_b|
MCE = max_b |accuracy_b − confidence_b|
Platt scaling fits a 1-D logistic regression on top of the raw predictions: p' = σ(a·logit(p) + b), with a, b found by gradient descent on the negative log-likelihood. It only rescales — it can't fix a non-monotone miscalibration.
Isotonic regression fits the best non-decreasing step function to (p, y) via the pool-adjacent-violators algorithm (PAVA): walk the points in order of p, and whenever an adjacent block's average would have to decrease to stay sorted, merge the two blocks into one (weighted average) and re-check. It is more flexible than Platt scaling but needs more data to avoid overfitting the bins.