Confusion Matrices, ROC Curves and AUC: Reading a Classifier Honestly

A threshold turns scores into decisions

Most binary classifiers do not output a hard yes/no — they output a continuous score (a probability, a distance, a logit), and a decision only appears once you pick a threshold: everything above it is called positive, everything below is called negative. Slide that threshold and every downstream metric changes, because you are not changing the model at all — you are changing where you draw the line through the same two overlapping score distributions, one for the true positives and one for the true negatives.

The four outcomes

Every prediction, compared against ground truth, falls into exactly one of four cells:

                  predicted positive     predicted negative
actual positive   True Positive  (TP)    False Negative (FN)
actual negative   False Positive (FP)    True Negative  (TN)

Every classification metric in common use is just some ratio of these four counts. Accuracy = (TP+TN)/total is the most obvious one and the least useful whenever the classes are imbalanced: a model that always predicts "negative" on a dataset that is 99% negative scores 99% accuracy while catching zero positives, which is exactly the failure mode that motivates every other metric below.

Precision, recall, and the trade-off between them

Precision = TP / (TP + FP)     "of everything I called positive, how much really was?"
Recall    = TP / (TP + FN)     "of everything that really was positive, how much did I catch?"
                                (recall is also called sensitivity or true positive rate)

F1 = 2 · Precision · Recall / (Precision + Recall)     harmonic mean of the two

Raising the threshold makes the classifier more conservative: fewer positives are declared, so precision tends to rise (the ones you do call positive are more likely to be right) while recall tends to fall (you miss more real positives by being cautious). Lowering the threshold does the opposite. This is not a bug to be optimised away — it is a real trade-off with no threshold-free solution, and which side of it matters more is a decision about the cost of the two error types, not a property of the model: a cancer screen wants very few missed positives (high recall, tolerating extra false alarms) while a spam filter wants very few real emails misclassified (high precision, tolerating some spam getting through).

The ROC curve and AUC

The receiver operating characteristic (ROC) curve sweeps the threshold through every possible value and plots the true positive rate (recall) against the false positive rate, FPR = FP/(FP+TN), at each one — tracing out the entire trade-off in one picture instead of one number per threshold:

TPR (recall)  = TP / (TP + FN)      y-axis
FPR           = FP / (FP + TN)      x-axis

diagonal TPR = FPR                  a classifier with zero information —
                                     equivalent to guessing at random
curve above the diagonal            genuine discriminative ability
top-left corner (0, 1)              the unreachable ideal: TPR=1, FPR=0

The area under the ROC curve (AUC) condenses the whole curve into one number between 0.5 (no better than random) and 1.0 (perfect separation), and it has a clean probabilistic reading that is worth memorising because it is the reason AUC is threshold-independent: AUC equals the probability that a randomly chosen true positive receives a higher score than a randomly chosen true negative. That is exactly why AUC does not care where you eventually set the threshold — it is measuring how well separated the two score distributions are in the first place, before any threshold decision is made.

When ROC is misleading: the precision-recall curve

ROC's false-positive rate is normalised by the count of true negatives, TN, which is a problem the moment negatives vastly outnumber positives — a huge absolute number of false positives can still be a tiny fraction of an enormous negative class, so FPR barely moves and the ROC curve looks deceptively good even while precision, which is normalised by the (much smaller) number of predicted positives instead, is collapsing. The precision-recall curve plots precision against recall directly and is the standard alternative whenever the positive class is rare — fraud detection, disease screening, rare-event detection — precisely the situations where ROC's optimism is most misleading.

Picking a threshold that isn't 0.5 by default

Nothing makes 0.5 the correct threshold except habit; it is only optimal when false positives and false negatives cost exactly the same and the classes are balanced, which is rarely true in practice. the Youden J statistic, J = TPR − FPR = TPR + TNR − 1, picks the threshold that maximises the vertical distance from the ROC curve to the diagonal — a reasonable default when both error types matter equally — but the honest way to choose a threshold is to attach an explicit cost to a false positive and a false negative and pick whichever threshold minimises the expected total cost, which is exactly what sliding the threshold across the two score distributions in the live demo lets you see happening directly, cell by cell, rather than through a single summary curve.

Frequently asked questions

Try it live

Everything above runs in your browser — open Confusion Matrix Explorer and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.