Test Reliability Lab: Item Difficulty, Measurement Error and Pass/Fail Mistakes

Presets

Controls

Readouts

Independent checks

The science

Each student has a hidden ability theta ~ N(0,1). Item i is answered correctly with probability from the two-parameter logistic (2PL) item response model with guessing floor g: P_i(theta) = g + (1-g) / (1 + exp(-1.7 a_i (theta - b_i))), where b is difficulty and a discrimination. The expected (true) score is T(theta) = sum_i P_i(theta).

Reliability from the observed 0/1 matrix is Kuder-Richardson 20 (Cronbach alpha for binary items): alpha = n/(n-1) (1 - sum p_i q_i / var(X)). Since the simulator knows theta, it can also compute the true reliability var(T)/var(X); alpha is a lower bound for it. The standard error of measurement is SEM = sd(X) sqrt(1 - alpha). Spearman-Brown predicts alpha for a test k times longer: alpha_k = k rho / (1 + (k-1) rho).

A pass/fail cut score turns a noisy score into a hard decision. Students whose true score is near the cut get misclassified; fewer and noisier items make this worse. Best accuracy needs items whose difficulty sits near the cut.

Simplifications: one latent trait, independent items, logistic (not normal-ogive) item curves, a fixed random cohort per seed. This is a measurement-theory demo, not advice on judging real students.