Each trial the judge scores response A and response B and picks a winner. The score fed to the judge is not the true quality alone — it mixes several terms:
score(X) = trueQuality(X)
+ posBias · [X shown first] · 0.5
+ verbBias · [X is verbose] · 0.4
+ noise, noise ~ N(0, σ² / K)
trueQuality(A) = 0 and trueQuality(B) = the quality-gap slider, so the ground-truth winner is fixed by that slider alone. Position bias rewards whichever response the judge happens to see first (order is randomised each trial). Verbosity bias always favours B, which is fixed as the longer response — a well-documented failure mode of LLM judges that has nothing to do with actual quality.
- Judges per trial (K) models self-consistency / ensemble sampling: averaging K independent noisy judgments shrinks the noise term by 1/√K, but does nothing to the systematic position and verbosity terms — they don't average away because they point the same direction every time.
- Counterbalance order evaluates the pair in both orders and averages the two scores per response. The position-bias term still appears in both averaged scores (once as "shown first", once as "shown second") — but as an equal constant added to A and B alike. Since the winner is decided by
scoreB − scoreA, an equal constant on both sides cancels out of that difference exactly, which is what actually matters — this is the one control that removes position bias from the decision rather than just averaging noise around it.
- Verbosity bias is not affected by either mitigation here, mirroring the real finding that length bias survives both ensembling and order-swapping unless the rubric explicitly penalises length.
Real-world relevance: "LLM-as-a-judge" pipelines (RLHF preference labeling, model leaderboards, automatic eval of chatbot responses) are known to inherit exactly these two biases from the underlying model, which is why production pipelines counterbalance presentation order and often add explicit length-normalisation to the rubric.
2D vs 3D: this page implements the identical trial model as the 3D version (same equations, verified algebraically — the counterbalanced position term is a common additive constant on both scores, so it cancels from the decision scoreB − scoreA even though it doesn't literally vanish from either score). The visualization trades the orbiting 3D towers for a flat particle-race arena plus a scrollable trend chart of accuracy / position-error over the last several hundred trials — drag the trend panel to scrub through history.