Forecaster Calibration Simulator

Forecaster Calibration

Overconfidence bias 0.30

Number of predictions 300

True event rate 0.50

Brier score

–

Calibration error (ECE)

–

Mean stated conf.

–

Actual hit rate

–

How it works

Each simulated event has a true probability p drawn around the "true event rate" you set. Our forecaster doesn't state p honestly — the overconfidence bias pushes their stated confidence further away from 50% than the truth warrants, so a coin-flip event gets reported as "80% likely" instead of "55% likely." The event then resolves as a real Bernoulli trial with probability p.

stated = 0.5 + (p − 0.5) · (1 + 2·bias)
Brier  = mean( (stated − outcome)² )
ECE    = Σ_bin (nᵢ/N) · |mean(statedᵢ) − mean(outcomeᵢ)|

Brier score — mean squared error between each stated probability and the 0/1 outcome. 0 is perfect, 0.25 is what "always say 50%" scores, 1 is worst possible.
Calibration error (ECE) — bins predictions by stated confidence (0–10%, 10–20%, …) and measures how far the actual hit rate in each bin strays from what was promised. A well-calibrated forecaster's "90% confident" bin is right ~90% of the time.
Overconfidence bias — 0 means the forecaster reports their true belief; higher values mean they round every probability toward the extremes, so their calibration curve bows below the diagonal on the confident end.
True event rate — the average true probability the underlying events are drawn around; it does not change the forecaster's honesty, only the mix of easy vs. hard calls.

This is the same math behind Philip Tetlock's forecasting-tournament research: tracking many probabilistic calls over time via Brier score and calibration curves is how "superforecasters" are separated from confident-sounding pundits.

Correct call Wrong call Calibration curve Perfect calibration