Research Power 3D

Two groups, one test statistic

Two groups of n participants have true means differing by dσ. With known σ the standard error of the difference of means is σ√(2/n), so the test statistic z = (meanT - meanC) / (σ√(2/n)) is N(0,1) if there is no effect (H0) and N(δ,1) with δ = d√(n/2) if there is one (H1). With two-sided level α the critical value is zc = Φ-1(1-α/2).

Power

Power is the probability that a real effect lands beyond the critical walls: 1 - Φ(zc - δ) + Φ(-zc - δ). It is the red part of the blue H1 curve; α is the orange part of the grey H0 curve. Each ball dropped on the track is one simulated study (its z value); the bars are the histogram of balls, drawn as a density so the shape settles as more studies finish, and the tower fill is the share of studies that were significant. The 80 % ring marks the usual planning target.

The crowds

The two crowds show the participants of the latest experiment. Each figure is one score (height = 1 + factor x score). The scores are drawn from N(0,1) and shifted so that the control mean is 0 and the treatment mean equals the experiment's observed difference, which is exactly the one behind the ball that just landed; the discs mark the group means. With a small n the observed difference wanders far from d.

Why low power also misleads

Among significant results the average estimated effect is larger than the true one (the readout "inflation of significant effects", computed from the positive significant simulated differences). Underpowered studies therefore both miss real effects and exaggerate the ones they find; planning n in advance, usually for 80 or 90 % power, is a core part of research methodology. Cohen's rule of thumb: d = 0.2 small, 0.5 medium, 0.8 large.

Independent numerical checks

Check 1: all M simulated experiments (same seed and draw order as the 2D page: the difference is drawn as d + σ√(2/n) · N(0,1)) must give a rejection rate within four binomial standard errors of the analytic power, and the displayed n for 80 % must really bracket the target (power at n80 at least 0.8, at n80-1 below 0.8). Check 2 is a different experiment: 1 000 trials in which n real random scores per group are generated with another seed and tested, which does not use the shortcut of drawing the difference directly.

Honest simplifications

Known variance (z-test, not t-test), equal group sizes, normal data, a single pre-planned test, no multiple comparisons, no publication filter beyond the inflation readout. For n above about 30 the t-test result is nearly identical. Figure heights are exaggerated for visibility.

Things to try

• Preset Small effect (d = 0.2): the H1 curve almost coincides with H0 and most balls are blue.
• Press Set n for 80 % power and watch the tower fill climb to the green ring.
• Lower alpha: the walls move out, alpha shrinks, and so does power.

Educational model; not a statistical-consulting tool.