Presets
Playback
Population and model
Synthetic people only: change base rates and how well the model separates each group.
Decision thresholds
Pick a fairness policy or set the two approval thresholds yourself.
Layers
Model checks
The science
One score, two groups
Every synthetic person has a hidden true outcome y (1 = would repay / succeed) and gets a model score. In each group g the scores are two bell curves: s | y=0 ~ N(0,1) and s | y=1 ~ N(dg,1), and a share pg of the group is truly positive (the base rate). Approving everyone with s ≥ tg gives, in closed form, TPR = 1 − Φ(t − d), FPR = 1 − Φ(t), selection rate p·TPR + (1−p)·FPR and precision PPV = p·TPR / selection. Sweeping t draws the ROC curve of each group; its area is AUC = Φ(d/√2), which the page checks against a trapezoid sum and against a rank-sum on the sampled people.
Four fairness ideas
Demographic parity asks for equal selection rates, equal opportunity for equal TPR, equal FPR together with equal TPR is called equalized odds, and predictive parity asks for equal PPV. The policy buttons solve for the group-B threshold that enforces each one while group A stays where you set it.
Why they cannot all hold
Writing the confusion matrix of any classifier in terms of p, PPV and FNR gives an identity (Chouldechova, 2017; Kleinberg, Mullainathan and Raghavan, 2016): FPR = p/(1−p) · (1−PPV)/PPV · (1−FNR). If two groups have different base rates p, and a classifier is not perfect, then equal PPV and equal FNR force different FPRs. The page evaluates the identity on both the formulas and the sampled confusion matrices (it is algebra, so it holds to rounding error) and the gap-against-threshold graph shows that no value of tB zeroes the selection, TPR, FPR and PPV gaps at once. Equal base rates and equal d are the special case where one threshold does satisfy several at once.
The utility line and calibration
Counting +1 for each correct approval and −c for each wrong one, the best threshold for a group is where the two bell curves weigh equally: t* = d/2 + ln(c(1−p)/p)/d. Because p and d differ by group, so does t*. The calibration graph is P(y=1 | s) = 1/(1 + exp(−(ln(p/(1−p)) + d·s − d²/2))): the same score means different odds in each group when base rates differ.
Honest simplifications: all people are synthetic and drawn from bell curves with equal variance; real scores are not binormal, real groups overlap, and labels carry their own bias (who was approved in the past decides who has a recorded outcome). The 1,600 people are one fixed random draw, so counts scatter by a few percent around the formulas, which is what the Monte-Carlo check measures. Utility counts only money-like gains and ignores harms that do not fit a single cost. The page shows what the mathematics permits; deciding which kind of fairness matters is a human and legal choice, not something a slider can settle.
What to try
- Press Equal opportunity and watch TPR match while FPR and PPV still differ; press Predictive parity and see selection and TPR split apart.
- Choose Equal groups: with equal base rates and equal separation, one threshold gives parity on every measure at once.
- Data gap lowers group B's AUC: its ROC curve sags and no threshold makes both groups equally good.
- Drag an amber threshold plate or line; in the graph of gaps, find where each curve crosses zero: they never meet at one point.
- Press Auto-sweep tB and watch the balance scale tilt with the selection gap.