Government agencies that pay out benefits, grants, or rebates increasingly use statistical risk scoring to decide which claims deserve a closer look. This simulation generates a synthetic population of several hundred claims with an 8% underlying fraud rate — deliberately low, because real-world benefit fraud is a genuinely rare event relative to the volume of legitimate claims — and trains a real logistic regression classifier on two visible features (claim-amount z-score and filing-pattern anomaly score) using batch gradient descent on the cross-entropy loss, exactly as a production model would be trained, just at a scale you can watch converge epoch by epoch.
Once trained, every claim carries a live fraud probability, rendered as a probability heatmap behind the scatter. Dragging the threshold slider changes which claims count as "flagged", and the confusion matrix, precision, recall, F1 score, and false-positive rate all update immediately against the population's true (normally hidden) labels. A review-capacity control adds the constraint every real fraud team operates under: investigators can only review so many claims per period, so the "statistically correct" threshold and the "operationally survivable" threshold are frequently two different numbers.
Why is precision so hard to achieve here, even with a well-separated model, when only 8% of claims are fraudulent?
This is the base-rate (prior) effect. Suppose 500 claims include 40 fraudulent ones (an 8% rate) and the model achieves a strong 85% recall with a modest 6% false-positive rate on the much larger legitimate group. It correctly flags 34 of the 40 fraud cases (true positives) — but the false-positive rate applies to all 460 legitimate claims, producing about 28 false positives. Precision = 34/(34+28) ≈ 55%, even though the underlying model is genuinely good. When the condition you are looking for is rare, the sheer size of the negative class means even a low false-positive rate generates an absolute number of false alarms that rivals the true positives. This is the same arithmetic behind the classic "rare disease test" paradox in medicine.
What does each cell of the confusion matrix mean operationally for a government fraud-review team?
True positive: a genuinely fraudulent claim is flagged, gets investigated, and the finding is confirmed — the system worked. False positive: a legitimate claimant is flagged, consumes an investigator's time on a case that turns out to be clean, and may face a stressful review, a payment delay, or an appeals process — this is the investigation burden and, at scale, an equity and trust cost. False negative: a fraudulent claim is scored below the threshold and receives no scrutiny at all — money is paid out that should not have been, and no one is ever notified to look again unless the case resurfaces some other way. True negative: a legitimate claim is correctly left alone — the invisible, uneventful, and most common outcome, since most claims are legitimate.
Why does review capacity matter as a real constraint on top of the statistical threshold choice?
A threshold chosen purely to balance precision and recall statistically can still flag far more claims than investigators can actually review in a period — turning a scoring problem into a triage problem. If capacity is 50 reviews per period but the threshold flags 140 claims, either the queue backs up indefinitely, cases sit unreviewed past useful deadlines, or staff make rushed, lower-quality decisions. In practice, agencies often invert the logic: instead of picking a probability cutoff first, they rank all claims by fraud score and take the top N up to capacity, which is mathematically equivalent to choosing whatever threshold happens to produce exactly N flags that period. Capacity, not statistics alone, frequently ends up setting the real-world operating threshold.
Card fraud detection scores a transaction in milliseconds and can auto-block or auto-approve it instantly, with a fast, low-stakes remedy if it gets it wrong (a card decline, a follow-up text). Government benefit-fraud screening instead feeds a slower human review pipeline, is subject to due-process and appeal rights, and a false positive can mean a vulnerable claimant's income is suspended or delayed for weeks while a case is investigated — a much higher cost of error on the people affected. Benefit fraud is also typically rarer as a share of claims, the features available are usually administrative rather than behavioural-biometric, and the legal and reputational consequences of a wrongful flag are far more consequential for the public sector than for a bank.
Features like claim amount and filing-pattern anomalies can correlate, indirectly, with protected characteristics such as household composition, disability, ethnicity, or postcode-linked deprivation — even when those characteristics are never used directly. If false positives fall disproportionately on already-disadvantaged groups, an automated screening tool can systematically compound harm against people least able to absorb a wrongful investigation or a suspended payment. Real-world cases such as the Netherlands childcare-benefits scandal and Australia's Robodebt scheme showed how automated fraud and debt-detection systems, deployed without adequate subgroup auditing, explainability, or a genuine right of appeal, caused large-scale wrongful harm. That is why subgroup-level precision/recall audits, human review of flagged cases, and clear appeal rights are treated as first-class requirements in this domain, not optional extras.
Logistic regression combines the two input features with learned weights w₁, w₂ and a bias b into a single score z = w₁x₁ + w₂x₂ + b, then squashes that score into a probability with the sigmoid function p = 1/(1+e⁻ᶻ). Training means choosing w₁, w₂, b to minimise cross-entropy loss — the average of −[y·log(p) + (1−y)·log(1−p)] over every claim, where y is the true fraud label. Because this loss is convex in the weights for logistic regression, gradient descent — repeatedly computing the gradient of the loss with respect to each weight and stepping a small distance against it — reliably converges to the same global optimum every time you retrain, which this simulation runs for real, epoch by epoch, watching the loss fall.
The model outputs a continuous fraud probability for every claim; the threshold slider decides the cutoff above which a claim counts as "flagged". Lowering the threshold flags more claims overall, which necessarily catches more of the true fraud cases (higher recall) but also sweeps in more legitimate claims that merely scored moderately high (lower precision). Raising the threshold does the reverse: fewer, more confident flags, so precision rises but recall falls as marginal fraud cases slip under the bar. Because the underlying score distribution overlaps between classes, no single threshold maximises both simultaneously — the slider lets you trace out that entire trade-off curve live, cell by cell, in the confusion matrix.