SERP Title A/B Test Power Lab

Two search-result titles, one impression stream: how many views do you need, and when is a CTR difference real?
Switch to 2D

Results per arm

1. Test statistic under H0 and H1 (power = green area)
2. Power vs impressions per arm
3. Evidence over time: -log10(p)
4. Observed CTR with 95 % interval
5. Bayesian posterior of each CTR (Beta)
6. Repeated experiments: p-value histogram
The science

What is modelled

A page can show one of several search-result titles. Every impression is an independent coin flip: the viewer clicks with probability p (the CTR). In arm i with n impressions the click count is Binomial(n, pi). Title A is the control with CTR p0; title B has a true relative lift L, so p1 = p0 (1 + L). Extra variants have zero true effect: they show what happens when you test many titles and pick the best-looking one.

Hypothesis test (two proportions)

z = (p1_hat - p0_hat) / sqrt( p_pool (1 - p_pool) * 2/n )
p-value = 2 * (1 - Phi(|z|))          reject H0 if p < alpha

H0: both titles have the same CTR. The p-value is the probability, if H0 were true, of a difference at least this large. It is not the probability that B is better.

Power and sample size

power(n) = Phi( (|d| sqrt(n) - z_(1-a/2) sqrt(2 pbar qbar)) / sqrt(p0 q0 + p1 q1) ) + (tiny opposite tail)
n = ( z_(1-a/2) sqrt(2 pbar qbar) + z_(power) sqrt(p0 q0 + p1 q1) )^2 / d^2      per arm
d = p1 - p0,  pbar = (p0+p1)/2,  q = 1 - p

Required n grows as 1/d2: halving the lift needs about four times the impressions. The minimum detectable effect (MDE) at a given n is found by inverting the same formula. Effect size is also shown as Cohen's h = 2 asin(sqrt p1) - 2 asin(sqrt p0).

Many variants

Testing k variants, each at level alpha, gives a family-wise error rate (chance of at least one false winner) of about 1 - (1 - alpha)^k if comparisons were independent; here they share one control, so the lab uses the exact normal-theory integral 1 - ∫ phi(x) [Phi(x+c) - Phi(x-c)]^k dx, c = z sqrt 2. Bonferroni tests each at alpha/k (safe but conservative); Holm steps down through the sorted p-values (never worse than Bonferroni).

Peeking

Checking the p-value at several looks and stopping at the first p < alpha is not one test but many. With alpha = 5 % and 5 equally spaced looks the real false-positive rate is about 14 % (tabulated by Armitage, McCray and Rowe, 1969). Fixed-horizon tests must be read once at the planned n; sequential designs need alpha-spending boundaries.

Bayesian view

posterior: p_i ~ Beta(a0 + clicks_i, b0 + n - clicks_i),  a0 = s p0, b0 = s (1 - p0)
P(B > A) = ∫ f_B(x) F_A(x) dx          expected loss = E[max(p_A - p_B, 0)]

Here s is the prior strength in pseudo-impressions centred on the baseline CTR. P(B > A) answers the question people usually mean; it is not a p-value and has its own early-stopping pitfalls.

Honest simplifications

Independent checks (bottom of the controls)

Normal CDF and inverse against table values; a textbook sample size; Monte Carlo power vs the closed form; Monte Carlo type-I error vs alpha; Bayesian P(B>A) and expected loss by quadrature vs Monte Carlo; family-wise error with and without Bonferroni vs the shared-control integral; and a 5-look peeking simulation vs the tabulated value (typical figure quoted from memory). Monte Carlo tolerances scale with 4 standard errors.

Lab bench modelled and Cycles-baked in Blender, drawn with Three.js r160 (WebGL). Maths identical to the 2D companion (assets/core.js).
Made with 🟠 Blender · Three.js