Presets
Playback
Forecaster and events
Layers
Model check
The science
What this is. A forecaster issues a probability f for a yes/no event (rain tomorrow); reality gives the outcome o in {0,1}. This lab scores forecasts the way meteorologists do. It is the honest, testable core of "can anyone predict the future": probabilistic forecasts either beat a naive baseline over many cases or they do not. The "Psychic" preset stands in for any claim of precognition: forecasts independent of the outcome. They show no skill at all. Nothing here simulates precognition.
Model
- Each event has a true chance p ~ Beta(μκ, (1-μ)κ); the outcome is Bernoulli(p). Small κ gives mostly near-certain events (high predictability).
- The forecaster sees p through a distorting lens: logit f = k logit p + b + σz, z ~ N(0,1). k=1, b=0, σ=0 is a perfect forecaster. Forecasts are rounded to 10 levels 0.05 ... 0.95.
- Brier score BS = mean (f-o)^2 (0 is perfect). Climatology UNC = o(1-o) with o the observed base rate: the score of always forecasting the base rate. Skill score BSS = 1 - BS/UNC; positive means better than climatology.
- Murphy decomposition BS = REL - RES + UNC, REL = (1/N) sum n_i (f_i - o_i)^2 (calibration error), RES = (1/N) sum n_i (o_i - o)^2 (ability to separate cases). Exact when forecasts take discrete values, as here.
- Reliability diagram: for each forecast level, how often the event really happened. Perfect = diagonal; overconfident forecasters give an S-shaped curve. Here the yellow bubbles are these observed frequencies with 95% Wilson intervals, the green diamonds the analytic expectation for infinite data.
- Bayesian updating. theta = probability that the forecaster's yes/no call (f ≥ 0.5) is right. With a flat Beta(1,1) prior and s hits in n calls the posterior is Beta(1+s, 1+n-s). "P(beats trivial)" is the posterior mass above max(μ, 1-μ), the accuracy of always saying the common answer.
Independent checks
- The simulated BS is compared with a deterministic quadrature over the Beta density of p and a Gaussian noise grid (within 3.5 standard errors).
- The Murphy identity must close to rounding error.
- The Beta posterior mean is compared with a numerical grid integral.
Simplifications
Events are independent and stationary; the "true chance" p is a modelling device that real forecasters never observe. The honest / overconfident / biased forecasters are illustrative rules, not measurements of any real service. One run is a random sample (fixed seed; Restart repeats it, a slider change starts a new run). The diagnosis text is a simple heuristic label (weighted slope of observed on forecast), not a formal test.