How Deep Learning Reads a Chest X-Ray for Signs of Pneumonia

A plain-language look at how convolutional neural networks and transfer learning with VGG16 detect pneumonia on chest X-rays, why sensitivity is prioritised over raw accuracy, and how Grad-CAM helps verify the model is looking in the right place.

▶ Open the simulation

Why chest X-rays are a natural fit for computer vision

Pneumonia remains one of the most common reasons for hospital admission worldwide, and the chest X-ray is usually the first imaging test a clinician orders when it's suspected. On a normal chest X-ray, healthy lung tissue appears mostly dark (air does not attenuate X-rays much), while pneumonia typically shows up as patches of increased opacity — whiter, cloudier regions — where infection has filled the airspaces with fluid and inflammatory cells. That's a genuinely visual pattern-recognition task, which is exactly what convolutional neural networks are built for: learning, directly from thousands of labelled example images, what the visual signature of "infected lung" looks like as opposed to "clear lung," without anyone having to hand-code the relevant radiological rules.

Building and comparing two different networks

A useful way to see what transfer learning buys you is to build two models side by side on the same chest X-ray dataset. The first is a custom CNN assembled from scratch: stacked Conv2D layers to extract local visual features, MaxPooling to progressively shrink the image while keeping the most important signals, batch normalisation to stabilise training, and dropout to reduce overfitting, all trained from a random starting point purely on the pneumonia dataset. The second uses VGG16, a well-known architecture pretrained on the 14-million-image ImageNet dataset, with its convolutional base kept and its final classification layers replaced and fine-tuned specifically for the normal-vs-pneumonia task. Because VGG16 already knows how to detect general-purpose visual features like edges, textures and shapes from its ImageNet training, it typically needs far less task-specific data to reach strong performance than a network starting from nothing — a genuinely important advantage in medical imaging, where large labelled datasets are expensive and slow to build.

Augmentation rules that respect anatomy

Standard image augmentation techniques — small rotations, brightness and contrast shifts, zoom — help a model generalise to X-rays taken with different equipment, exposure settings and slightly different patient positioning, effectively multiplying the size and diversity of the training set without collecting a single new image. But not every augmentation makes sense for a chest X-ray: horizontal flipping is acceptable since the chest is roughly left-right symmetric, but vertical flipping is deliberately avoided, because an upside-down chest X-ray is not a physiologically realistic image and would just teach the model a distorted, unhelpful version of lung anatomy. This kind of domain-aware constraint is a good example of why medical imaging pipelines can't just borrow generic computer vision recipes wholesale — the augmentation strategy has to respect what's anatomically and clinically plausible.

Why sensitivity is the metric that actually matters

Once trained, a battery of metrics gets reported: accuracy, precision, recall (sensitivity), F1-score, specificity, and the area under the ROC curve. In a medical screening context these are not interchangeable, and recall deserves particular attention. If a patient with real pneumonia is missed (a false negative), treatment gets delayed — a genuinely serious clinical consequence. If a healthy patient is flagged for a closer look that turns out to be unnecessary (a false positive), that's an inconvenience and a use of clinical time, but a far less serious error. This asymmetry is why, in diagnostic screening, teams deliberately tune models to favour high sensitivity even if that costs a little bit of specificity, and why a single accuracy number, reported on its own, can be actively misleading about how safe a model would be to deploy.

Grad-CAM: checking the model is looking at the lungs

A high sensitivity score alone doesn't prove a model has learned anything medically meaningful — it could, in principle, be picking up on some unrelated artefact, like a hospital-specific marker printed on the image, that happens to correlate with the pneumonia label in the training data. Grad-CAM addresses this by tracing the gradients flowing back from the model's final prediction through its last convolutional layer, producing a heatmap that highlights exactly which pixels most influenced the decision. For a trustworthy pneumonia classifier, that heatmap should consistently light up over the lung fields themselves, particularly the areas of opacity a radiologist would point to — not the corners of the image, not text overlays, not the diaphragm. This kind of explainability check is also increasingly expected by regulators: transparency around how an AI reaches its diagnostic conclusions is treated as a prerequisite for clinical trust, not an optional extra.

What this can and can't do for the NHS

Realistically, a model like this — trained on one public dataset from a single hospital in Guangzhou — is a solid demonstration of the underlying technique but far from something ready to support NHS radiology directly. Any credible clinical use would sit as a decision-support layer: helping triage and prioritise a radiologist's reading list in a busy department, or providing a rapid first-pass read in resource-constrained settings, always with a human radiologist making the final call. It would also need retraining and validation on data collected across multiple UK hospitals and imaging systems, since a model trained on one population and one set of X-ray machines can perform noticeably worse when it meets data from elsewhere — a well-documented failure mode in medical AI generally.

Frequently Asked Questions

What does pneumonia actually look like on a chest X-ray?

Areas of the lung affected by pneumonia typically appear as patches of increased opacity, or whiteness, compared to the normally dark, air-filled lung tissue, because infection fills the small airspaces with fluid and inflammatory cells that attenuate X-rays more than air does.

Why compare a custom CNN against a transfer learning model like VGG16?

It illustrates a core lesson in medical imaging AI: pretrained networks like VGG16 already know general-purpose visual features from a huge unrelated dataset, so they usually reach strong accuracy with much less task-specific training data than a network built entirely from scratch.

Why does sensitivity matter more than overall accuracy for pneumonia screening?

Because missing a real case of pneumonia (a false negative) delays treatment and carries much higher clinical risk than a false alarm that simply prompts a closer look. Models tuned purely for accuracy can quietly trade away sensitivity without that trade-off being obvious from the headline number.

What does Grad-CAM add beyond a performance score?

It shows visually which parts of the X-ray drove the model's decision, letting clinicians and developers check that the model is actually focusing on lung pathology rather than an irrelevant artefact that happened to correlate with the label in the training data.