Home▸AI & Machine Learning▸A/B Testing Pitfalls: Peeking, Sample Ratio Mismatch, and When to Use a Bandit Instead

🧪 A/B Testing Pitfalls: Peeking, Sample Ratio Mismatch, and When to Use a Bandit Instead

Sample-size calculation, the peeking problem, multiple-testing correction, and sample ratio mismatch — the statistical rigor that separates a trustworthy A/B test from a costly false positive, plus when Multi-Armed Bandits are the better tool.

AI & Machine Learning3DAdvanced60 FPS

Simulated visitors fall through a router into two 3D silos — Variant A and Variant B — converting or not based on each arm's true probability, while a live dashboard tracks the running z-test, its p-value trace, and a sample-ratio-mismatch check.

🔬 What It Demonstrates

Peeking at results after every arrival inflates the real false-positive rate far past the nominal 5%; a router bug that skews traffic away from the configured split gets caught by a chi-squared test long before the conversion-rate difference means anything.

🎮 How to Use

Set B's true lift and the configured traffic split, then toggle continuous peeking, a router bug, or switch the traffic router to a Thompson-sampling bandit to see how each choice changes what the dashboard reports.

💡 Did You Know?

Sample ratio mismatch checks use very strict thresholds (χ² p far below 0.001) precisely because even a tiny routing bug reliably produces a detectable imbalance once enough users have been split — no need to wait for a big one.

⚙ Under the hood

machine learninga/b testingstatisticsdata analysisexperiment designsample sizeThree.js

3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install