Forecast calibration diagram (2D)

Score probability forecasts: reliability, Brier skill and a Bayesian test of whether a forecaster beats chance.

Switch to 3D

Presets

Playback

Forecaster and events

Layers

Model check

The science

What this is. A forecaster issues a probability f for a yes/no event (rain tomorrow); reality gives the outcome o in {0,1}. This lab scores forecasts the way meteorologists do. It is the honest, testable core of "can anyone predict the future": probabilistic forecasts either beat a naive baseline over many cases or they do not. The "Psychic" preset stands in for any claim of precognition: forecasts independent of the outcome. They show no skill at all. Nothing here simulates precognition.

Model

  • Each event has a true chance p ~ Beta(μκ, (1-μ)κ); the outcome is Bernoulli(p). Small κ gives mostly near-certain events (high predictability).
  • The forecaster sees p through a distorting lens: logit f = k logit p + b + σz, z ~ N(0,1). k=1, b=0, σ=0 is a perfect forecaster. Forecasts are rounded to 10 levels 0.05 ... 0.95.
  • Brier score BS = mean (f-o)^2 (0 is perfect). Climatology UNC = o(1-o) with o the observed base rate: the score of always forecasting the base rate. Skill score BSS = 1 - BS/UNC; positive means better than climatology.
  • Murphy decomposition BS = REL - RES + UNC, REL = (1/N) sum n_i (f_i - o_i)^2 (calibration error), RES = (1/N) sum n_i (o_i - o)^2 (ability to separate cases). Exact when forecasts take discrete values, as here.
  • Reliability diagram: for each forecast level, how often the event really happened. Perfect = diagonal; overconfident forecasters give an S-shaped curve. Here the yellow bubbles are these observed frequencies with 95% Wilson intervals, the green diamonds the analytic expectation for infinite data.
  • Bayesian updating. theta = probability that the forecaster's yes/no call (f ≥ 0.5) is right. With a flat Beta(1,1) prior and s hits in n calls the posterior is Beta(1+s, 1+n-s). "P(beats trivial)" is the posterior mass above max(μ, 1-μ), the accuracy of always saying the common answer.

Independent checks

  • The simulated BS is compared with a deterministic quadrature over the Beta density of p and a Gaussian noise grid (within 3.5 standard errors).
  • The Murphy identity must close to rounding error.
  • The Beta posterior mean is compared with a numerical grid integral.

Simplifications

Events are independent and stationary; the "true chance" p is a modelling device that real forecasters never observe. The honest / overconfident / biased forecasters are illustrative rules, not measurements of any real service. One run is a random sample (fixed seed; Restart repeats it, a slider change starts a new run). The diagnosis text is a simple heuristic label (weighted slope of observed on forecast), not a formal test.

Running Brier score vs analytic expectation
Posterior of the yes/no accuracy
Murphy decomposition
Made with 🟠 Blender