Once a classifier makes decisions about people -- loan approvals, hiring shortlists, content moderation -- "is it accurate" stops being the only question worth asking. "Is it fair" turns out to have several different, sometimes incompatible, mathematical definitions, and knowing which one your system optimises for (often implicitly, by default) matters as much as knowing its accuracy.
Two common fairness definitions
Demographic parity
Requires that the overall approval rate be equal across groups, regardless of whether the people approved are actually qualified. Easy to measure, easy to explain, but says nothing about whether the model is actually finding the right people within each group.
Equal opportunity
Requires that among people who are genuinely qualified (truly positive), the approval rate is equal across groups -- a narrower, arguably more meaningful notion of fairness for many high-stakes decisions, since it focuses on not under-serving qualified people rather than on raw approval counts.
💡 Key idea: when two groups have genuinely different underlying base rates, a single shared decision threshold generally cannot satisfy both demographic parity and equal opportunity at the same time -- this is a mathematical result, not a modelling failure.
Why base rate differences matter so much
If historical data reflects real disparities upstream of the model -- unequal access to credit history, unequal opportunity, biased historical labelling -- then the two groups' score distributions will genuinely differ even for a technically well-calibrated model. A single threshold applied to two different distributions will, in general, produce different approval rates and different true-positive rates simultaneously, and there is no single cutoff that zeroes out both gaps except in special cases.
The threshold trade-off
Using a separate threshold per group can close one gap (say, equalising true-positive rate) while widening the other (approval-rate parity) -- and using explicitly different thresholds per group is itself legally contested in many jurisdictions, since it can be read as disparate treatment even when the intent is to correct disparate impact. There is no context-free right answer; the correct trade-off depends on the specific legal, ethical and business setting the system operates in.
What this means in practice
- Decide which fairness definition matters most for the specific decision before building the model, not after auditing it.
- Report multiple fairness metrics, not just one -- a model can look fair on one axis and badly unfair on another.
- Treat a fairness gap as a prompt to investigate the data pipeline, not just the model -- the gap may originate upstream of anything the model controls.
- Involve people with domain and legal expertise in the trade-off decision; it is not a purely technical choice.
🧪 Try it yourself: the AI Ethics Lab simulation lets you experiment with everything described above directly in your browser.