400 synthetic loan applicants are generated with a hidden protected attribute g ∈ {A,B} and a legitimate feature F (real creditworthiness) drawn independently of g — the two groups are equally creditworthy in reality. But the historical training labels the bank actually has on file are not the truth: they were produced by decades of biased human loan officers, so group B's observed historical approval label is suppressed relative to its real merit:
fair label fairY = 1 if F ≥ 0 else 0 (real creditworthiness — never observed by the model)
hist. label P(histY=1) = sigmoid(1.5·F − 2.4·corr·[g=B]) (what the bank's own records say — biased)
A "neutral" proxy feature P (e.g. a zip-code index) is built from group membership, the legitimate feature and noise:
P = corr · sign(g) + signal · F + noise · ε
70% of applicants train the model, the other 30% are held out for the audit below.
A real 2-input logistic-regression classifier is fit with batch gradient descent (300 epochs, cross-entropy loss, L2 penalty) on standardized [P, F] against the historical label — g is never a model input. Raising Proxy↔group correlation does two real things at once, exactly as historical redlining did: it makes the training labels more biased against group B and makes P a sharper stand-in for g, so gradient descent has both the incentive and the means to lean on P. The trained weight on P (see "Learned weight on proxy") is not fabricated — it is read straight off the fitted model — and it is what drives the disparity below even though the model has literally never seen g.
The audit panel evaluates the trained model on the 30% held-out test individuals and computes the legal adverse-impact / four-fifths rule (US EEOC guidance, echoed in EU AI Act Art. 10/9 data-governance and risk-monitoring duties):
selection rate SR(g) = approvals(g) / applicants(g)
disparate-impact ratio = min(SR_A, SR_B) / max(SR_A, SR_B)
rule of thumb: ratio ≥ 0.80 → passes; below → adverse impact flag
Test accuracy compares the model's approve/deny call against the hidden fair label (real creditworthiness, F ≥ 0) on the held-out set — not against the biased historical label it was trained on. That gap is the whole point: a model can be "accurate" by the bank's own biased records while being measurably wrong, and unfair, against the truth — the only number a real auditor cannot see but this sandbox can.