🧪 A/B Testing Pitfalls Lab

About this simulation

Simulated visitors fall from the hopper, get routed by the splitter into Variant A (slate) or Variant B (red), and convert or bounce based on each arm's true conversion probability. Converted visitors turn green and stack up inside their silo; the live dashboard tracks the running two-proportion z-test, its p-value, and a chi-squared sample-ratio-mismatch check.

How it works

• Peeking — with the peek toggle on, the test is checked for significance after every single arrival. Because early samples are noisy, this repeatedly gives you many independent chances to see p<0.05 by pure luck — inflating the true false-positive rate far above 5%, even when the true lift is zero.
• Sample ratio mismatch (SRM) — the router bug skews actual traffic away from the configured split (e.g. a caching or redirect bug favouring one arm). A chi-squared goodness-of-fit test compares observed vs. expected counts; a very small χ² p-value (≪0.001) is a red flag that something upstream of the statistics is broken, and the whole test should be distrusted regardless of its p-value.
• Fixed-horizon testing — turn peeking off and the dashboard estimates keep updating live, but the honest verdict is only trustworthy once you've committed to a sample size in advance and stopped exactly once.
• Multi-armed bandits — switching the router to Thompson sampling reallocates traffic toward whichever arm currently looks better, using a Beta posterior per arm. This trades a clean p-value for lower regret (fewer visitors sent to the losing variant) — useful when the goal is maximizing conversions during the test itself, not producing a publishable p-value.

Controls

• True lift — B's real conversion-rate advantage over A's ~12% baseline (can be negative or zero).
• Configured split — the traffic share intended for Variant A.
• Traffic router — fixed random split vs. an adaptive Thompson-sampling bandit.
• Peek toggle — check significance continuously vs. only trust the estimate at a fixed horizon.
• Router bug toggle — injects a sample ratio mismatch to see how the χ² check catches it.

Did you know?

Continuous peeking with a naive 5% threshold can push the real false-positive rate above 30–40% given enough looks — one of the most common ways A/B tests ship a "winning" change that was actually noise.