A page can show one of several search-result titles. Every impression is an independent coin flip: the viewer clicks with probability p (the CTR). In arm i with n impressions the click count is Binomial(n, pi). Title A is the control with CTR p0; title B has a true relative lift L, so p1 = p0 (1 + L). Extra variants have zero true effect: they show what happens when you test many titles and pick the best-looking one.
z = (p1_hat - p0_hat) / sqrt( p_pool (1 - p_pool) * 2/n ) p-value = 2 * (1 - Phi(|z|)) reject H0 if p < alpha
H0: both titles have the same CTR. The p-value is the probability, if H0 were true, of a difference at least this large. It is not the probability that B is better.
power(n) = Phi( (|d| sqrt(n) - z_(1-a/2) sqrt(2 pbar qbar)) / sqrt(p0 q0 + p1 q1) ) + (tiny opposite tail) n = ( z_(1-a/2) sqrt(2 pbar qbar) + z_(power) sqrt(p0 q0 + p1 q1) )^2 / d^2 per arm d = p1 - p0, pbar = (p0+p1)/2, q = 1 - p
Required n grows as 1/d2: halving the lift needs about four times the impressions. The minimum detectable effect (MDE) at a given n is found by inverting the same formula. Effect size is also shown as Cohen's h = 2 asin(sqrt p1) - 2 asin(sqrt p0).
Testing k variants, each at level alpha, gives a family-wise error rate (chance of at least one false winner) of about 1 - (1 - alpha)^k if comparisons were independent; here they share one control, so the lab uses the exact normal-theory integral 1 - ∫ phi(x) [Phi(x+c) - Phi(x-c)]^k dx, c = z sqrt 2. Bonferroni tests each at alpha/k (safe but conservative); Holm steps down through the sorted p-values (never worse than Bonferroni).
Checking the p-value at several looks and stopping at the first p < alpha is not one test but many. With alpha = 5 % and 5 equally spaced looks the real false-positive rate is about 14 % (tabulated by Armitage, McCray and Rowe, 1969). Fixed-horizon tests must be read once at the planned n; sequential designs need alpha-spending boundaries.
posterior: p_i ~ Beta(a0 + clicks_i, b0 + n - clicks_i), a0 = s p0, b0 = s (1 - p0) P(B > A) = ∫ f_B(x) F_A(x) dx expected loss = E[max(p_A - p_B, 0)]
Here s is the prior strength in pseudo-impressions centred on the baseline CTR. P(B > A) answers the question people usually mean; it is not a p-value and has its own early-stopping pitfalls.
Normal CDF and inverse against table values; a textbook sample size; Monte Carlo power vs the closed form; Monte Carlo type-I error vs alpha; Bayesian P(B>A) and expected loss by quadrature vs Monte Carlo; family-wise error with and without Bonferroni vs the shared-control integral; and a 5-look peeking simulation vs the tabulated value (typical figure quoted from memory). Monte Carlo tolerances scale with 4 standard errors.