The Class-Imbalance Trap: Why Accuracy Lies in Rare-Disease Machine Learning

A 95% accurate model that is completely useless

Here is a genuinely instructive failure case from applied medical machine learning. Take a dataset of 5,110 patients where 4,861 never had a stroke and 249 did — a roughly 19.5-to-1 imbalance between the majority and minority classes. Now build the laziest possible 'model': one that always predicts 'no stroke', regardless of any input. Measured on plain accuracy, that model scores 95.1%. It sounds impressive on a slide. It is also worthless, because it will fail to identify a single one of the 249 patients who actually had a stroke. In a screening context, that is the one group of patients the entire exercise exists to find.

This is not a hypothetical trap; it is the exact baseline used in a real stroke-risk modelling project, deliberately included in the results table specifically to demonstrate the problem before more sophisticated models are introduced. It is one of the most instructive numbers in the whole project, because it shows in one line why accuracy is the wrong headline metric whenever the outcome you care about is rare.

Why class imbalance is the rule, not the exception, in medicine

Rare positive outcomes are the norm across clinical prediction, not an unusual edge case. Stroke, specific cancers, sepsis, rare genetic conditions and adverse drug reactions are all, thankfully, uncommon in the general population — which is exactly why they are hard to catch and exactly why predictive screening tools are valuable in the first place. But that same rarity means any dataset built from real-world or public health records will be dominated by negative cases, and any model trained naively on that data will gravitate toward the lazy strategy of predicting the majority class most of the time, because doing so minimises average error across the whole training set even though it is catastrophic for the minority class specifically.

This creates a direct tension between what a naive optimisation process rewards and what a clinical deployment actually needs. A model optimising for overall accuracy on a 19.5:1 imbalanced dataset can improve its accuracy score simply by becoming more conservative and predicting negative more often — the opposite of what a screening tool should do.

Precision, recall and why F1 is the tie-breaker

Two metrics matter more than accuracy here: precision, the proportion of patients flagged as high-risk who actually were high-risk, and recall, the proportion of genuinely high-risk patients the model successfully caught. These two numbers naturally trade off against each other — a model can trivially achieve 100% recall by flagging every patient as high risk, at the cost of terrible precision, or 100% precision by only flagging the single most obvious case, at the cost of terrible recall. F1-score is the harmonic mean of the two, and it was explicitly chosen as the primary evaluation metric for the stroke project precisely because it punishes models that sacrifice one metric to inflate the other, forcing a genuine balance.

On the held-out test set, the best-performing tuned XGBoost model achieved precision of 78% and recall of 84%, combining to an F1-score of 0.80 — a very different, far more informative picture than the 95% accuracy of the do-nothing baseline. Reading the underlying confusion matrix makes the trade-off concrete: out of 1,022 test patients, the model correctly flagged 14 true stroke cases while missing 37, and raised 54 false alarms among patients who did not go on to have a stroke. The deliberate lean toward higher recall over precision (84% vs 78%) reflects a considered clinical choice: in a screening context, missing a real stroke risk (a false negative) is generally judged more harmful than one extra unnecessary follow-up appointment (a false positive).

Fixing the imbalance during training, not just measuring it afterwards

Choosing F1 as the scoring metric only solves half the problem — it correctly evaluates a model, but does nothing on its own to make a model better at the minority class. The other half of the solution happens during training. The stroke project used class weighting: for logistic regression, a 'balanced' class-weight setting that automatically up-weights errors on the minority class during optimisation; for XGBoost, an explicit scale_pos_weight parameter set to 19.5, matching the actual imbalance ratio in the training data, which tells the algorithm to treat a missed stroke case as roughly nineteen and a half times more costly than a false alarm during training.

The measured effect of this reweighting was substantial: class weighting improved F1-score by roughly 0.05 to 0.10 across the different model families tested, a meaningfully large jump for a single hyperparameter change. Other established techniques for the same underlying problem include oversampling methods like SMOTE (Synthetic Minority Oversampling Technique), which generates synthetic examples of the rare class to rebalance the training set, and undersampling the majority class — both listed as promising directions for future iterations of this particular project, alongside ensemble stacking of multiple model types.

A checklist for spotting this problem in the wild

The pattern generalises well beyond stroke prediction, and is worth recognising whenever you encounter a machine-learning result reported for a medical, fraud-detection, safety, or any other rare-event domain. First, check the class balance of the underlying dataset before trusting any accuracy figure — a headline accuracy above roughly 90% on a dataset where the positive class is known to be rare is a signal to look further, not a reason for confidence. Second, look specifically for precision, recall and F1 (or an equivalent balanced metric) rather than accuracy alone; if only accuracy is reported for an imbalanced problem, treat the result with real scepticism. Third, check whether the training process did anything explicit to address the imbalance — class weighting, resampling, or a cost-sensitive loss function — since a model trained without any such adjustment will tend to underperform on the minority class regardless of which metric is used to report it afterwards. None of this requires advanced statistics to check; it just requires asking one question before accepting any accuracy number at face value: how rare is the thing we're actually trying to detect?