Home▸Computer Science▸The Peeking Problem: A/B Test Significance (2D)

The Peeking Problem: A/B Test Significance (2D)

Interactive 2D simulator of the optional-stopping bias in mobile A/B tests: watch synthetic users flow into control/variant funnels on a live canvas, a two-proportion z-test p-value strip-chart update every batch, and repeated peeking inflate the true false-positive rate above the nominal alpha.

Computer Science2DAdvanced60 FPS📱 Mobile-adapted⇄ 3D version

Synthetic users stream into a control funnel and a variant funnel at rates you choose, a live two-proportion z-test tracks the p-value after every batch, and a growing strip-chart shows exactly when the result crosses the significance threshold. Switch between "peek every batch" and a pre-committed fixed sample size to watch, over dozens of repeated experiments, why checking a dashboard every day and stopping the moment it turns green inflates the real false-positive rate far above the advertised α — the optional-stopping bug behind a lot of unreliable mobile A/B test results.

⚙ Under the hood

Watch synthetic users flow into control and variant funnels on a live 2D canvas while a two-proportion z-test tracks the p-value, and see how repeatedly peeking at significance inflates the true false-positive rate above the nominal alpha.

A/B testingstatisticsp-valuehypothesis testingmobile analyticssequential testing

2D · HTML5 Canvas 2D · 60 FPS target · runs fully client-side, no install