The Bias-Variance Tradeoff: Why Every Model Must Choose Its Mistakes

Two very different ways for a model to be wrong

Every predictive model makes errors, but not all errors come from the same source. Imagine training the same kind of model on several different random samples drawn from the same underlying population — say, several different surveys of house prices in the same city. A model with high bias will tend to make the same kind of systematic mistake regardless of which sample it was trained on: it might consistently underestimate prices in expensive neighbourhoods no matter which houses happened to be in its particular training sample, because the model itself is too simple to represent that relationship at all. A model with high variance, by contrast, produces wildly different predictions depending on the specific sample it happened to be trained on — a slightly different set of houses in the training data leads to a noticeably different, and often much less reliable, model.

Bias is error from wrong assumptions baked into the model — the difference between what the model can represent and what the true underlying relationship actually looks like. Variance is error from excessive sensitivity to the particular quirks and noise of one training sample. A useful shorthand: high bias means the model is consistently missing the target in the same direction; high variance means the model's shots are scattered all over the target, inconsistent from one attempt to the next.

Underfitting and overfitting are the visible symptoms

Bias and variance are the underlying statistical causes; underfitting and overfitting are what you actually observe when you evaluate a model. An underfit model (high bias) performs poorly even on the data it was trained on, because it is too simple to capture the real pattern — fitting a straight line to a relationship that is fundamentally curved is a classic example. Both training accuracy and validation accuracy will be disappointingly low, and, crucially, similar to each other, because the model's limitation is structural rather than a matter of memorising noise.

An overfit model (high variance) does the opposite: it achieves excellent, sometimes near-perfect, accuracy on its training data, because it has enough flexibility to memorise the specific examples it saw — including their noise, outliers, and coincidental patterns that don't actually generalise. When evaluated on new, unseen validation data, performance drops noticeably. This gap between training performance and validation performance is the single most useful diagnostic signal in applied machine learning: a wide gap points to overfitting (variance), while uniformly poor performance on both sets points to underfitting (bias).

Why you cannot minimise both at once

The word "tradeoff" is doing real work here: reducing bias and reducing variance generally pull a model in opposite directions. Increasing a model's complexity — adding more features, more polynomial terms, more layers, more decision-tree depth — lets it represent more intricate relationships, which reduces bias, since the model is now flexible enough to represent the true underlying pattern more faithfully. But that same added flexibility also gives the model more room to fit noise specific to the training sample, which increases variance.

Plotted against increasing model complexity, training error falls continuously — a sufficiently complex model can always fit its own training data better, right down to memorising it perfectly. Validation error, however, traces out a U-shaped or J-shaped curve: it falls at first, as increasing complexity helps the model capture genuine patterns, reaches a minimum at some "sweet spot" of complexity, and then rises again as additional complexity starts fitting noise rather than signal. The total expected error a model makes on new data can be decomposed, mathematically, into three additive pieces: bias squared, variance, and irreducible noise inherent to the problem itself (measurement error, genuinely random factors) that no model, however good, can eliminate. The practical goal of model selection is finding the point on that complexity axis where the sum of bias and variance is smallest — not zero bias, not zero variance, but the best available balance.

Diagnosing where a model sits

Plotting learning curves — training and validation error as a function of the amount of training data used — is one of the most direct ways to diagnose whether a model is suffering more from bias or variance. If both curves converge to a similarly poor error level as more data is added, and adding more data does not close the gap, the model is underfitting; the fix is to increase model complexity (more features, a more flexible algorithm) rather than collect more data. If the training curve stays low while the validation curve remains substantially higher, even as more data is added, the model is overfitting; here, additional training data, regularisation, or a simpler model architecture are more likely to help than adding features.

Cross-validation, where a dataset is repeatedly split into different training and validation portions and results are averaged, gives a much more reliable read on this gap than a single train/validation split, particularly for smaller datasets where a single unlucky split could give a misleading picture of where the model actually sits on the complexity spectrum.

Practical tools for shifting the balance

Once you have diagnosed which side of the tradeoff a model is struggling on, a range of standard techniques can shift it back toward the sweet spot. To reduce bias (fight underfitting): use a more flexible model class, add more informative features, reduce regularisation strength, or train for longer. To reduce variance (fight overfitting): gather more training data, apply regularisation (L1/L2 penalties that discourage overly large weights, or dropout in neural networks), reduce model complexity (fewer features, shallower trees, fewer layers), use ensemble methods that average across many models to cancel out individual variance, or stop training early once validation performance stops improving. Recognising that these levers pull in opposite directions on the bias-variance seesaw — and choosing the right one for the symptom actually observed, rather than applying a generic fix — is one of the most valuable diagnostic skills in practical machine learning work.