Why Doctors Won't Trust a Black Box: SHAP and Explainable AI in Healthcare

A number alone is not an answer

Imagine a risk model tells a GP that a patient has a 78% probability of stroke risk within a defined horizon. On its own, that number is close to useless to a clinician. It gives no indication of which of the patient's characteristics are driving the score, whether the model is picking up on something modifiable (like uncontrolled blood glucose) or something fixed (like age), and whether the prediction is being driven by a sensible clinical pattern or a statistical artefact of the training data. A doctor asked to act on that number — perhaps by starting a new medication, or referring for further tests — is being asked to trust a black box, and clinicians are, rightly, reluctant to do that with patient care.

This is the core motivation behind explainable AI (XAI) in medicine: predictions need to come with reasons attached, in a form a domain expert can check against their own knowledge. SHAP, short for SHapley Additive exPlanations, has become one of the most widely used tools for this purpose in tabular healthcare models, precisely because it produces a reason for every single prediction, not just an average description of what the model tends to care about.

How SHAP actually works, in plain terms

SHAP is built on an idea borrowed from cooperative game theory: the Shapley value, which fairly divides a 'payout' among players based on their individual contribution across every possible combination of players. Applied to a machine-learning model, the 'game' is the model's prediction for one specific patient, the 'players' are that patient's individual feature values (age, glucose, hypertension status and so on), and the 'payout' being divided is the gap between the model's prediction for this patient and the model's average prediction across all patients.

Concretely, for a stroke-risk model this means every feature value gets its own signed number for a given patient: a positive SHAP value pushes the predicted risk up, a negative one pulls it down, and the sizes are directly comparable and additive — they sum, along with a baseline value, to the model's actual output for that patient. This is what makes SHAP different from older techniques like plain feature-importance rankings, which only describe a model's behaviour in aggregate. SHAP gives you both: a global view (which features matter most across the whole population) and a local view (which features drove this specific person's score).

A worked example: two very different patients

The value of local explanations becomes obvious when you compare two contrasting cases from a tuned XGBoost stroke model. For a 72-year-old patient with an average glucose level of 180 mg/dL and existing hypertension, the model produced a high predicted risk, and SHAP attributed roughly +0.35 of that prediction to age alone, +0.28 to the elevated glucose reading, and +0.15 to hypertension — together pushing the score well above the decision threshold. For a 28-year-old patient with a glucose level of 85 mg/dL and no hypertension, the same three features contributed negatively (around -0.22, -0.18 and -0.12 respectively), pulling the prediction down to a low-risk classification.

Neither number, on its own, tells a clinician anything new — a doctor already knows that a 72-year-old with hypertension and high glucose is higher risk than a healthy 28-year-old. What SHAP adds is precision and auditability: it quantifies exactly how much each factor contributed for this individual, in units that are comparable across features, so a clinician can spot-check whether the model is reasoning sensibly (age and glucose dominating, as expected) or oddly (for example, if marital status were somehow driving half the prediction, that would be a red flag worth investigating before trusting the model further).

Explainability as a bias detector, not just a trust exercise

Beyond building clinician confidence, SHAP analysis doubles as a diagnostic tool for the model itself. In the stroke-risk case, the global SHAP ranking showed engineered features like an 'is senior' age-threshold flag and a diabetic glucose category appearing in the top ten most influential variables, which validated that the feature-engineering choices made during preprocessing were actually adding predictive signal rather than noise. But the same analysis also surfaced features that deserve more scrutiny: occupation type and marital status showed measurable, if smaller, contributions to predicted risk. These variables likely act as proxies for socioeconomic status and lifestyle factors not directly captured in the dataset, and a model leaning on them risks encoding structural health inequalities — for instance, systematically scoring patients in certain occupations as higher risk for reasons that have more to do with data correlations than individual physiology.

This is precisely why documentation practices like model cards explicitly flag socioeconomic and demographic bias as an open question requiring further testing rather than a solved problem, even for a model with a respectable overall F1-score. Explainability does not eliminate bias, but it makes bias visible enough to investigate — a black-box model gives you no such handle.

The limits of explainability

SHAP is a powerful diagnostic, but it is not a substitute for clinical validation, and it is worth being clear-eyed about what it cannot do. It explains what the model learned from its training data, not what is objectively true about disease mechanisms — if the training data itself is biased or unrepresentative of the UK population, SHAP will faithfully explain a biased model without flagging that the underlying data was the problem. It also does not establish causation: a large positive SHAP value for glucose level tells you the model relies heavily on glucose to make its prediction, not that lowering a specific patient's glucose will proportionally lower their real-world stroke risk, even though that broader relationship is well established in the clinical literature. Used well, SHAP is one part of a larger governance process that should also include external validation on independent, ideally UK-representative datasets, explicit bias testing across demographic subgroups, and ultimately sign-off from clinical experts before any tool like this moves anywhere near patient-facing use.