Rubric Design & Inter-Rater Reliability Simulator

Rubric Design

Descriptor specificity 50%

Performance levels 4

Number of graders 5

Grading Outcome

Inter-rater agreement

—

Mean deviation

—

Class composite

—

Score noise σ

—

gold ring = true (target) level
blue spheres = each grader's score

How it works

Every criterion in a rubric has a hidden "true" quality level for a given work sample — the level a perfectly calibrated grader would assign. A real grader's actual score deviates from that true level by random error, and the size of that error depends on how precisely the level descriptors are written:

score = true_level + N(0, σ),  clipped to [1, L]
σ = σ_max · (1 − specificity)
σ_max scales with the number of levels L

Vague descriptors ("good use of evidence") leave more room for interpretation, so σ is large and graders scatter across levels. Specific, behaviourally-anchored descriptors ("cites ≥3 sources, each explicitly tied to the thesis") shrink σ toward zero, so independent graders converge on the same score.

Descriptor specificity — how precisely each level is defined; drives the scatter you see around the gold target ring.
Performance levels — more levels (finer granularity) also means more ways to disagree, widening σ_max.
Number of graders — more independent raters give a more stable read on true agreement.
Inter-rater agreement — the fraction of grader pairs whose scores land within one level of each other, the simulator's stand-in for statistics like Cohen's κ or ICC used in real rubric validation.

Real-world relevance: this is exactly why rubric-writing guides insist on concrete, observable criteria — a vague rubric doesn't just confuse students, it makes two teachers grading the same essay disagree.

Target level Grader score

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.