Home▸AI & Machine Learning▸Inter-Annotator Agreement Simulator (Cohen's Kappa)

Inter-Annotator Agreement Simulator (Cohen's Kappa) — 2D

Interactive 2D simulator of dataset labeling: two independent raters label the same items, a live contingency-matrix heatmap plus Cohen's Kappa vs raw percent agreement compare in real time as you tune item count, class balance, rater accuracy and item ambiguity.

AI & Machine Learning2DModerate60 FPS📱 Mobile-adapted⇄ 3D version

Dataset governance lives or dies on label quality, and label quality is measured, not assumed. This 2D counterpart puts two independent raters on the same batch of items, each governed by a personal accuracy and each item carrying its own hidden ambiguity, then renders every item as a tile colored by what actually happened: green where both raters agree and are right, amber where they agree and are both wrong, red where they disagree outright. A live 2×2 contingency-matrix heatmap and a raw-percent-agreement-vs-Kappa comparison bar make the chance-correction visible in real time — push class balance to an extreme and watch percent agreement stay flattering while Cohen's Kappa reveals how little of that agreement survives correcting for chance.

⚙ Under the hood

Two independent raters label the same dataset in 2D — tune rater accuracy, item ambiguity and class balance and watch observed agreement, chance agreement and Cohen's Kappa update live, the statistic real labeling pipelines use to certify dataset label quality.

machine learningdata labelingdataset governancecohen's kappaannotation qualitystatistics

2D · HTML5 Canvas 2D · 60 FPS target · runs fully client-side, no install