Classroom Trial: Measuring a Teaching Intervention's Effect Size

Trial Design

Students per group 30

True effect size (Cohen's d) 0.50

Score variability (SD points) 10

Significance threshold α

Trial Results

Control Treatment

t(58.0) = 0.00, p = 1.000

Observed Cohen's d

0.00

p-value

1.000

Decision at α

—

Est. power (a priori)

How it works

Each "Run New Trial" draws two independent samples of exam scores — a control group with no intervention and a treatment group that received a teaching intervention — from normal distributions with the population standard deviation you set:

Control:   X_c ~ N(70, σ²)
Treatment: X_t ~ N(70 + d·σ, σ²)

where d is the true (population) Cohen's d you dial in with the effect-size slider — this is the ground truth the experiment is trying to detect; the observed d recovered from the sampled data will differ trial to trial by chance.

The two groups are then compared with Welch's t-test (unequal-variance safe), whose exact two-tailed p-value comes from the regularized incomplete beta function — no lookup table, no approximation:

t  = (x̄_t − x̄_c) / √(s_t²/n_t + s_c²/n_c)
df ≈ Welch–Satterthwaite degrees of freedom
p  = I_{df/(df+t²)}(df/2, 1/2)     (two-tailed)

Observed Cohen's d uses the pooled sample standard deviation:

d = (x̄_t − x̄_c) / s_pooled,   s_pooled = √[((n_t−1)s_t² + (n_c−1)s_c²) / (n_t+n_c−2)]

The estimated power is the a-priori probability of detecting the true effect at the chosen α, using the standard normal-approximation formula for an equal-n two-sample test:

power ≈ Φ( |d_true| · √(n/2) − z_(1−α/2) )

This is exactly the logic behind classroom-level randomized controlled trials in educational research (e.g. comparing a new teaching method against business-as-usual): random assignment removes selection bias, the sample size and expected effect size jointly determine whether a trial is even capable of detecting a real effect, and a "non-significant" result with a small sample and small true effect is often just an under-powered trial, not proof the intervention doesn't work.

Control mean Treatment mean

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.