Each glowing point is one dataset item. Every tick, a batch of items is sent to N simulated raters, each of whom votes with some probability of flipping the true label — controlled by rater noise. A majority vote decides the consensus label; items where the winning share falls below the consensus threshold are marked as disagreements. A slice of items are secretly gold questions with a known correct answer — used to catch systematically biased raters, exactly as described in the article's "Labeling Program for Safety" example.
agreement per item p_i = votes_for_majority / N
observed agreement p̄ = mean(p_i)
expected by chance p_e ≈ 1 / classes
Cohen's-κ estimate κ = (p̄ − p_e) / (1 − p_e)
- Raters per item — how many independent annotators label each data point (more raters → more stable consensus, higher labeling cost).
- Rater noise — probability an individual rater mislabels an item, modeling inconsistent guidelines or fatigue.
- Gold-question audit rate — the fraction of items with a known ground truth, used to flag raters and items for review.
- Consensus threshold — the minimum vote share required before a label is accepted instead of routed to adjudication.
Green points have reached confident consensus, amber points are under the threshold and routed to adjudication, and red-ringed points are gold questions whose consensus label disagreed with the known answer — the audit signal that drives rater retraining.