Beyond a Single CNN: Ensembling, Uncertainty and Threshold Tuning for Pneumonia X-Ray Screening

Why one CNN is not enough

A convolutional neural network that spots pneumonia on a chest X-ray is now a familiar demonstration project. What is far less familiar — and far more relevant to whether such a tool could ever sit inside a real triage workflow — is what happens once you stop trusting a single model's output at face value. A single network, however well trained, has three separate weaknesses: its accuracy is capped by the particular visual biases it happened to learn, it gives you no signal about when it is guessing rather than confident, and its default 0.5 decision boundary was never chosen with any clinical consequence in mind. This article looks at a research project that tackles all three problems at once, using a chest X-ray pneumonia dataset as the working example, by combining model ensembling, Monte Carlo Dropout uncertainty quantification and Youden's J threshold optimisation into a single pipeline. None of these techniques are exotic on their own, but stacked together they turn a bare classifier into something closer to a clinically usable screening instrument — one that reports not just "pneumonia" or "normal" but how sure it is, and applies a different confidence bar depending on what the prediction will be used for.

Four architectures, one weighted vote

The project trains four separate transfer-learning models on the same chest X-ray dataset (5,216 training images, 624 held out for testing, resized to 224×224), each built on a different ImageNet-pretrained backbone with its convolutional base frozen and a small custom classification head (global average pooling, two dense layers with dropout, and a sigmoid output) trained on top: VGG16, ResNet50, MobileNetV2 and EfficientNetB0. Evaluated individually, the four land at noticeably different points: VGG16 reaches 89.2% accuracy with an AUC of 0.94, ResNet50 improves on that at 90.5%/0.95, MobileNetV2 — a network designed for mobile efficiency rather than peak accuracy — actually comes in slightly below the VGG16 baseline at 88.7%/0.93, and EfficientNetB0 leads the individual models at 91.1%/0.96.

None of those numbers is the headline result. The interesting step is what happens when the four models' output probabilities are combined via a weighted average rather than picked from in isolation. Because each architecture makes somewhat different errors — a texture pattern that fools ResNet50 doesn't necessarily fool EfficientNetB0, and vice versa — averaging their probability outputs (weighted toward the stronger individual performers) cancels out a portion of each model's idiosyncratic mistakes. The ensemble reaches 92.3% accuracy and a 0.97 AUC, a +3.1 percentage point improvement over the best single architecture. That gain looks modest as a bare number, but in a binary screening task operating in the 90%+ accuracy range, each additional point represents a shrinking pool of genuinely hard, ambiguous cases — exactly the ones where a second and third architectural "opinion" is most likely to catch something one network alone would miss. Simple majority voting is a coarser alternative to weighted averaging; the project reports both were tested, with weighted averaging of the continuous probability outputs producing the better result because it preserves more information than a binary vote from each model.

What the ensemble is actually correcting for

Beyond raw accuracy, the project reports the fuller diagnostic picture for the ensemble: 96.8% sensitivity (correctly flagging pneumonia when it is present), 87.2% specificity (correctly clearing normal scans), 89.5% precision, 95.7% negative predictive value, and an F1-score of 93.1%. It also reports two metrics that matter more in a clinical evaluation than in a typical machine-learning benchmark: Cohen's Kappa at 0.88, which measures agreement with ground-truth labels after correcting for the agreement you'd expect from chance alone (0.88 sits in the "excellent agreement" range rather than merely "good"), and the Matthews Correlation Coefficient at 0.86, a metric that stays informative even when a dataset's two classes are imbalanced, unlike plain accuracy. Two likelihood ratios round out the picture: a positive likelihood ratio (LR+) of 15.6 means a positive prediction is over fifteen times more likely to come from a true pneumonia case than a true normal case, and a negative likelihood ratio (LR-) of 0.04 means a negative prediction sharply reduces the probability that pneumonia is actually present. Ratios of that magnitude, in the classic Sackett interpretation used in evidence-based medicine, are considered to generate large, often decisive, shifts in diagnostic probability — which is precisely why an ensemble's small accuracy gain over a single network can matter clinically even though it looks unremarkable on a leaderboard.

Monte Carlo Dropout: making the network say "I'm not sure"

A standard neural network trained with dropout only uses that dropout during training, as a regularisation trick to prevent overfitting; at inference time dropout is normally switched off so the network gives a single, deterministic prediction. Monte Carlo Dropout deliberately breaks that convention: dropout layers are left active during inference, and the same X-ray image is pushed through the network dozens of times, with a different random subset of neurons silenced on each pass. Because each pass effectively samples a slightly different sub-network, the collection of resulting probability outputs behaves like an approximate Bayesian posterior over the model's belief — a distribution rather than a point estimate.

The practical payoff is a per-image uncertainty score sitting alongside the probability itself. Two X-rays can both receive a "70% likely pneumonia" prediction from the point-estimate model, yet one might produce a tight cluster of MC Dropout passes (say, consistently between 65–75%) while the other scatters wildly (anywhere from 20% to 95% across different passes). The first case is a confident call; the second is a network essentially shrugging. In a screening context this distinction is operationally critical: it is the mechanism behind routing roughly 75% of cases to fully automated reporting while flagging the remaining 25% — the ones where the model's internal "opinion" is unstable — for mandatory review by a radiologist. That confidence-based triage split is exactly how the project frames its automation rate, and it only works because MC Dropout gives you a variance to threshold on, not just a probability to threshold on.

Youden's J: picking the threshold, not just the model

Even a perfectly calibrated, ensembled, uncertainty-aware model still needs a decision boundary — some probability value above which the output is reported as "pneumonia." The default choice, 0.5, treats a false positive (telling a healthy patient they may have pneumonia) and a false negative (missing a real case) as equally costly. In an emergency-department triage setting they are not: sending a healthy patient for an unnecessary follow-up chest exam is an inconvenience, while missing an actual pneumonia case delays antibiotic treatment. Youden's J statistic, defined simply as J = sensitivity + specificity − 1, gives a principled way to search across every possible threshold on the ROC curve and identify the one that maximises the combined true-positive and true-negative rate — the point on the curve furthest from the diagonal line representing random guessing.

Rather than settling on one universal threshold, the project derives three, each tuned to a different clinical use case by weighting sensitivity or specificity differently: a screening threshold of 0.30 for A&E rapid triage, which pushes sensitivity up to 98.2% at the cost of specificity dropping to 65.4% — appropriate when the goal is "don't miss anything, we'll sort out the false alarms later." A diagnostic threshold of 0.45, close to the Youden's J optimum for balanced classification, yields 96.8% sensitivity and 87.2% specificity as a general-purpose operating point. And a confirmation threshold of 0.70, used before committing a patient to treatment, deliberately trades sensitivity down to 85.3% in exchange for specificity up to 95.1% — minimising the chance of treating someone unnecessarily. The key idea is that "the model's threshold" is not a fixed property of the network; it is a policy decision that should move with the cost of the two error types in whatever step of the clinical pathway the prediction is feeding into.

How the three techniques combine into one pipeline

None of ensembling, MC Dropout, or Youden's J threshold selection solves the deployment problem in isolation. Together, they describe a coherent pipeline: the ensemble of four CNNs produces a more reliable base probability than any single architecture could; MC Dropout runs that ensemble's prediction (or each member model) through repeated stochastic passes to attach a confidence band to the number; and Youden's J-derived thresholds decide, given the intended use of that specific prediction — rapid A&E triage versus pre-treatment confirmation — where the line between "flag it" and "clear it" should sit. A prediction that lands in high-confidence territory on a low-stakes screening threshold can be auto-reported; a prediction near a threshold boundary, or one where MC Dropout shows wide disagreement across passes, gets escalated to a radiologist regardless of which side of the line the point estimate falls on. That combination is also what the project's Grad-CAM visualisations are for: for any case a clinician does review, Grad-CAM heatmaps highlight which regions of the X-ray most influenced the network's decision, giving a visual sanity check rather than asking the reviewer to trust an opaque number. The project also runs 5-fold cross-validation with reported confidence intervals to check that the accuracy and AUC figures are stable across different train/test splits rather than an artefact of one lucky split, and exports the final ensemble to TFLite (a 74% size reduction, from 59MB down to 15MB) and ONNX for lower-latency, cross-platform inference — relevant because a triage tool that takes minutes per image defeats the purpose regardless of how good its statistics are.

What this is, and what it explicitly is not

It is worth being precise about the status of a project like this. It is a research and educational pipeline built on a well-known public Kaggle dataset of paediatric chest X-rays, not a validated clinical product. The gap between "reaches 92.3% accuracy and a 0.97 AUC on a held-out test split of one dataset" and "safe to use on real NHS patients" is large and well defined in UK regulation: software making diagnostic claims about a specific patient would need to be classified and approved as a medical device by the MHRA (very likely Class IIa given the diagnostic function), would need multi-site validation across more than one NHS Trust to confirm the results generalise beyond one dataset's particular imaging equipment and patient population, and would need to pass through the NHS Data Security and Protection Toolkit and a formal ethics review before touching real patient data. The 92.3%/0.97 AUC figures, and the illustrative £2.5M/year cost-saving projection sometimes attached to workload-automation estimates like these, are best read as "here is what a well-executed transfer-learning and ensembling pipeline can achieve on a public benchmark," not as evidence a tool is ready for a hospital corridor. The techniques covered here — ensembling, uncertainty quantification, and cost-sensitive threshold tuning — are precisely the kind of methodological groundwork that a real clinical AI submission would need to demonstrate, but they are the beginning of that regulatory and validation process, not the end of it.