SAST Triage Gate: Precision, Recall and Base Rate

A conveyor of findings: lateral position is the scanner score, the orange fence is the gate. Red balls are real vulnerabilities, blue ones are false alarms. Drag to orbit.

Switch to 2D
Controls

Presets

Parameters

Quality

Camera

Model checks

The science

Signal detection. A static analyser gives every finding a score. Real vulnerabilities score N(d, 1), false alarms N(0, 1); d (d-prime) measures how well the tool separates them. A gate flags a finding when its score exceeds t, so recall = TPR = Φ(d - t) and FPR = Φ(-t). The ROC curve plots TPR against FPR as t sweeps; its area is Φ(d/√2), the chance that a random real finding outscores a random false alarm.

Base rate. What a developer feels is precision, from Bayes: precision = π·TPR / (π·TPR + (1-π)·FPR). When real vulnerabilities are rare (small π) even a good scanner produces mostly false alarms. That is the base-rate fallacy, and the reason “99 % accurate” tools still drown teams in noise.

Choosing the gate. With a review cost of 1 unit per false alarm and cm units per missed vulnerability the cost per finding is C(t) = π cm (1 - TPR) + (1 - π) FPR. Setting dC/dt = 0 gives t* = d/2 + ln[(1-π)/(π cm)] / d: a stricter gate when bugs are rare, a laxer one when a miss is expensive. The chart shows the numerical minimum agrees.

Simplifications. Equal-variance Gaussian scores (real tools have skewed, discrete scores and per-rule behaviour); findings are independent; every flagged finding is reviewed correctly; the cost of a miss is one number although it depends on severity and exploitability. Defensive triage teaching only. Checks: ROC area by integration vs closed form, the optimal threshold by numerical minimisation vs closed form, a Monte Carlo of 40,000 findings against Bayes, and a rank test with random pairs.

Loading model...
Loading Blender model...
-
-
Made with 🟠 Blender · Three.js