500 synthetic samples each get a true latent probability q ~ Uniform(0,1) and a label y ~ Bernoulli(q). A classifier's raw predicted probability is a distorted version of q:
p_raw = σ( k · logit(q) + noise )
logit(q) = ln( q / (1−q) )
k = 1 would be a perfectly calibrated classifier. k > 1 makes it overconfident (pushes probabilities toward 0 or 1 faster than the data justifies) — the classic failure mode of deep nets and boosted trees. k < 1 makes it underconfident (predictions huddle near 0.5).
Reliability diagram: predictions are sorted into B equal-width bins by predicted probability. For each bin b:
confidence_b = mean predicted p in bin b
accuracy_b = fraction of true positives in bin b
A perfectly calibrated model has confidence_b = accuracy_b for every bin — the diagonal ramp in the scene. The blue bars (confidence) and orange bars (accuracy) diverging from it is exactly what miscalibration looks like.
Brier score = (1/N) Σ (p_i − y_i)² (lower is better)
ECE = Σ (n_b/N) · |accuracy_b − confidence_b|
MCE = max_b |accuracy_b − confidence_b|
Platt scaling fits a 1-D logistic regression on top of the raw predictions: p' = σ(a·logit(p) + b), with a, b found by gradient descent on the negative log-likelihood. It only rescales — it can't fix a non-monotone miscalibration.
Isotonic regression fits the best non-decreasing step function to (p, y) via the pool-adjacent-violators algorithm (PAVA): walk the points in order of p, and whenever an adjacent block's average would have to decrease to stay sorted, merge the two blocks into one (weighted average) and re-check. It is more flexible than Platt scaling but needs more data to avoid overfitting the bins.