Baseline conversion is fixed at 10% for A; B's true rate is A + the lift slider (hidden from the "test", exactly like a real experiment). Every visitor is routed to an arm, converts with that arm's true probability, and the running two-proportion z-test decides significance.
z = (p̂B − p̂A) / sqrt(p̄(1−p̄)(1/nA+1/nB))
SRM: χ² = Σ (observed − expected)² / expected, flag if χ² > 10.83 (p<0.001)
Thompson: sample θ ~ Beta(1+conversions, 1+failures) per arm, serve the higher draw
- Fixed 50/50 + Peek — checks p every few visitors and stops the instant p<0.05. With no real effect this false-positives far more than 5% of runs.
- Fixed 50/50 + Wait — only looks once the pre-planned sample size (≈3,838/variant, from a 2pp MDE at α=0.05, power=80%) is reached — the honest way to run a fixed test.
- Traffic-split bug — biases routing away from 50/50 without changing what's "expected"; the χ² sample-ratio-mismatch check catches it as a randomization bug, not a result.
- Bandit — Thompson Sampling reallocates traffic toward whichever arm looks best, trading a clean one-time p-value for fewer visitors wasted on the loser.
Each run auto-resets after it concludes so you can watch the false-positive rate and early-stop rate accumulate across many independent experiments.