A/B Testing Pitfalls: Peeking, Sample Ratio Mismatch, and When to Use a Bandit Instead

Why "try it and see" is not an experiment

A/B testing gets treated, casually, as if it were self-evidently rigorous simply because it involves splitting traffic and comparing two numbers. It is not automatically rigorous. Every step between "we changed something" and "the change is real and worth keeping" is a place where a poorly designed test can produce a confident, wrong answer — and because a wrong answer here often looks statistically clean (a p-value under 0.05, a plausible-sounding uplift number), it is far more dangerous than an obviously broken experiment. The gap between a casual A/B test and a trustworthy one is entirely in a handful of specific, learnable failure modes.

Sample size has to be calculated before the test starts, not after

The required sample size for a proportion-based test (a conversion rate, a click-through rate) depends on four quantities that have to be chosen deliberately: the baseline rate, the minimum detectable effect (MDE) — the smallest true difference worth caring about — the significance level (commonly α = 0.05), and the desired statistical power (commonly 80%, meaning an 80% chance of detecting a real effect of that size if it exists). Plugging a 10% baseline conversion rate and a 2-percentage-point MDE into the standard sample-size formula for two proportions yields roughly 3,800 observations per variant, or about 7,700 total — a number that, at 1,000 visitors a day, takes just under eight days to accumulate.

Skipping this calculation and instead running a test "until it feels done" produces two failure modes: stopping too early, before there is enough data to distinguish a real effect from noise, or running indefinitely past the point where the answer was already clear, wasting traffic that could have gone to a better-performing variant. Choosing the MDE honestly matters too — setting it artificially small to make a desired effect "detectable" just inflates the required sample size without changing whether the effect is actually there.

The peeking problem: why checking early inflates false positives

The single most common way a well-designed test gets undermined in practice is peeking: checking the p-value repeatedly as data accumulates and stopping the moment it first crosses the significance threshold. This feels harmless — surely checking more often just means noticing the result sooner — but it is a well-documented statistical error. A p-value threshold of 0.05 controls the false-positive rate for a single look at a fixed sample size; checking repeatedly gives the noise multiple chances to randomly cross that threshold even when there is no real effect, which can push the effective false-positive rate several times higher than the nominal 5%.

There are two legitimate ways around this, not one workaround. The first is discipline: calculate the required sample size in advance and do not look at significance until that sample size is reached. The second is a genuinely different statistical framework — sequential testing methods such as the Sequential Probability Ratio Test, which are explicitly designed to allow continuous monitoring and early stopping while still controlling the overall false-positive rate, at the cost of somewhat more complex implementation. What is not legitimate is checking a standard fixed-sample-size test every day and stopping "whenever it looks significant," which is the peeking problem in its most common disguise.

Multiple testing: the more metrics you check, the more false positives you manufacture

A related trap appears when a single experiment is evaluated against several metrics at once — conversion rate, average order value, time on page, return visit rate — and any one of them crossing significance is treated as a win. With a 5% false-positive rate per metric, testing five independent metrics gives roughly a 23% chance that at least one shows "significance" purely by chance, even if nothing about the experiment actually worked. The standard correction is to tighten the significance threshold in proportion to the number of metrics tested — a Bonferroni correction divides the target α by the number of tests, so five metrics at an overall 0.05 significance level require each individual metric to clear p < 0.01 before being called significant.

The more robust discipline, though, is architectural rather than statistical: define a single primary metric before the test launches, and treat everything else as a secondary or diagnostic metric that informs interpretation but does not by itself justify calling the test a win. This avoids both the multiple-testing inflation and the temptation to retroactively declare whichever metric happened to move in the right direction as "the metric that mattered."

Sample ratio mismatch: the bug detector most teams skip

Sample ratio mismatch (SRM) is a check that catches a different class of problem entirely — not a statistical subtlety but an outright implementation bug. If an experiment is configured to split traffic 50/50 but the actual observed split ends up meaningfully skewed (say 3,850 versus 4,200), a chi-square goodness-of-fit test against the expected ratio, checked with a strict threshold (commonly p < 0.001, tighter than the usual 0.05 because the cost of a false alarm here is low and the cost of missing a real bug is high), flags the mismatch. An SRM is a strong signal that randomization logic is broken — a caching layer serving one variant preferentially, a redirect that fails silently for one arm, a bot-filtering rule that disproportionately affects one variant — and any result from an experiment with an unresolved SRM should be discarded rather than interpreted, because the two groups are no longer comparable.

When a bandit beats a fixed A/B test

Traditional fixed-allocation A/B testing deliberately sends equal traffic to a variant that might be clearly worse, for the entire duration of the test, in exchange for a clean statistical comparison. Multi-Armed Bandit algorithms, such as Thompson Sampling, instead update their traffic allocation continuously as evidence accumulates, shifting more traffic toward whichever variant is currently performing best while still exploring the others enough to keep learning. In a simulated three-variant scenario with true conversion rates of 10%, 12%, and 11%, Thompson Sampling converges on favoring the 12% variant within a few thousand observations and captures measurably more total conversions over the run than an equal-traffic split would have, precisely because it stops wasting traffic on the weaker variants once the evidence is strong enough.

The trade-off is that a bandit is the wrong tool for a one-time, high-stakes launch decision that needs a clean, auditable statistical answer — a bandit's traffic allocation is itself the output of an ongoing statistical process, which makes "was variant B significantly better" a fuzzier question to answer after the fact than a fixed-sample-size test gives you. Bandits earn their keep in continuous optimization contexts with many variants and no single "launch moment" — content ranking, ad creative selection, pricing experiments running indefinitely — where minimizing the ongoing cost of testing matters more than producing a single, formally reportable p-value.