Home▸Computer Science▸The Peeking Problem: A/B Test Significance

The Peeking Problem: A/B Test Significance

Interactive 3D simulator of the optional-stopping bias in mobile A/B tests: watch synthetic users stream into control/variant buckets, a two-proportion z-test p-value trace update live, and repeated peeking inflate the true false-positive rate above the nominal alpha.

Computer Science3DAdvanced60 FPS📱 Mobile-adapted⇄ 2D version

Synthetic users stream into a control bucket and a variant bucket at rates you choose, a live two-proportion z-test tracks the p-value after every batch, and a growing 3D trace shows exactly when the result crosses the significance threshold. Switch between "peek every batch" and a pre-committed fixed sample size to watch, over dozens of repeated experiments, why checking a dashboard every day and stopping the moment it turns green inflates the real false-positive rate far above the advertised α — the optional-stopping bug behind a lot of unreliable mobile A/B test results.

⚙ Under the hood

Watch synthetic users stream into control and variant buckets while a live two-proportion z-test tracks the p-value, and see how repeatedly peeking at significance inflates the true false-positive rate above the nominal alpha.

A/B testingstatisticsp-valuehypothesis testingmobile analyticssequential testing

3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install