Inter-Annotator Agreement Simulator (Cohen's Kappa) — 2D

Dataset

Items to label 180

Class balance (share of class B) 50%

Raters

Rater accuracy 85%

Item ambiguity 30%

Agreement

0.00

Cohen's κ

Observed agreement pₒ

Chance agreement pₑ

Both raters agree, correct Both raters agree, wrong Raters disagree

How it works

Each item i has a hidden true label and a difficulty d_i ∈ [0,1]. A rater's chance of labeling it correctly blends personal accuracy a with the item's difficulty:

p_correct(i) = clamp(a − ambiguity·d_i, 0.5, 0.99)
label_i = true_i with prob p_correct(i), else flipped

Two raters label every item independently with fresh noise. Every labeled pair is tallied into a real 2×2 contingency matrix (rendered as the heatmap on the right), and both agreement statistics are read straight off that matrix — nothing is looked up or shortcut:

M[a][b] = count of items where rater A said a, rater B said b
p_o = (M[0][0] + M[1][1]) / N
P_A(1) = (M[1][0]+M[1][1])/N,  P_B(1) = (M[0][1]+M[1][1])/N
p_e = P_A(0)·P_B(0) + P_A(1)·P_B(1)
κ   = (p_o − p_e) / (1 − p_e)

κ < 0 — worse than chance (raters actively disagree).
κ ≈ 0–0.20 — slight agreement · 0.21–0.40 — fair · 0.41–0.60 — moderate
κ ≈ 0.61–0.80 — substantial · 0.81–1.00 — almost perfect (Landis & Koch scale)
Push class balance toward an extreme (e.g. 90%) at a fixed accuracy/ambiguity and watch the comparison bar: raw percent agreement barely moves, but κ drops — the majority class inflates po by chance alone, and only κ corrects for it.
Raising ambiguity pulls κ down even at high individual accuracy — this is why real annotation pipelines track κ, not just per-rater accuracy, and route low-κ item clusters to adjudication.

Real-world relevance: this is the same statistic (Cohen's/Fleiss' κ) used to certify human-labeled training data before it enters a model pipeline — the mechanic behind "label quality" in dataset governance.

⚠ Couldn't load the 2D engineCanvas rendering failed. Check your browser and reload.