3D Forecaster Reliability Diagram

3D Reliability Diagram

Overconfidence bias 0.30

Number of predictions 300

True event rate 0.50

Brier score

–

Calibration error (ECE)

–

Mean stated conf.

–

Actual hit rate

–

How it works

Predictions are grouped into ten stated-confidence bins (0–10%, 10–20%, … 90–100%). Each bin rises as a bar whose height is the actual fraction of events in that bin that really happened. A translucent diagonal reference plane marks perfect calibration — where bar height should sit if "80% confident" really meant right 8 times in 10. Bars that fall short of the plane on the confident end expose overconfidence directly as a gap you can see and rotate around.

bin height = mean(outcome | stated ∈ bin)
Brier = mean( (stated − outcome)² )
ECE   = Σ_bin (nᵢ/N) · |mean(statedᵢ) − heightᵢ|

Bar height — the real hit rate inside that confidence bin, drawn from live Bernoulli trials.
Reference plane — the diagonal surface where height = bin midpoint; toggle it off to inspect the bars alone.
Overconfidence bias — pushes stated probabilities toward the extremes without changing the true probabilities, so at high bias the tall "I'm 95% sure" bars visibly sag below the plane.
Bar width — scaled by how many predictions landed in that bin, so thin bars are low-sample and noisier.

This reliability-diagram view is the standard visualization from forecasting science (Tetlock's calibration/resolution decomposition of the Brier score) for auditing a single forecaster's track record, distinct from crowd-averaging methods that pool many forecasters together.

Well calibrated bin Overconfident bin Reference plane

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.