Every batch adds users to control (A) and variant (B) with true conversion probabilities pA and pB = pA + Δ. After each batch a two-proportion z-test is run on the accumulated counts:
p̂ᴬ = convA / nA, p̂ᴮ = convB / nB
p̂ = (convA + convB) / (nA + nB) (pooled proportion)
z = (p̂ᴮ − p̂ᴬ) / √( p̂(1−p̂)·(1/nA + 1/nB) )
p-value = 2·(1 − Φ(|z|)) (two-sided normal)
The strip chart on the right plots −log₁₀(p-value) against sample size; the red dashed line marks the α threshold — anything poking above it reads as "statistically significant" at that instant. The two funnel columns fill with the live conversion rate for each arm, and every user is drawn as a dot falling into its column, green if it converted.
- Peek every batch — the experiment stops the moment p first dips below α, exactly like a dashboard someone checks daily and stops as soon as it turns green. Because you're really taking a minimum over many correlated look-in-times, the true probability of a false alarm is far higher than the nominal α — this is the well-known optional-stopping / peeking problem in sequential testing.
- Fixed n = 3000 — the correct baseline: the sample size is committed in advance and the p-value is read exactly once, at the end. Its false-positive rate should track α closely.
- Cumulative stop rate — every finished experiment (early stop or reaching the cap) is logged; with Δ = 0 this is an empirical estimate of the real false-positive rate for the selected monitoring mode, shown as a bar next to the nominal α in the bottom panel.
Real-world relevance: this is exactly why mobile A/B testing platforms (Firebase Remote Config, Optimizely, Split.io) either fix the sample size up front or use dedicated sequential-testing corrections (e.g. always-valid p-values) — glancing at "is it significant yet" every day and stopping on the first green result routinely doubles or triples the advertised false-positive rate.