Why irrelevant features hurt, not just fail to help
With a limited number of training samples, a model has enough free parameters to fit spurious correlations between irrelevant features and the label that exist only by chance in that particular sample. Those spurious fits don't generalize — they actively degrade test-set performance, which is the counterintuitive core of the curse of dimensionality: more dimensions relative to sample size make overfitting easier and generalization harder.
Univariate ranking: a fast, simple filter
Univariate feature selection scores each feature independently — typically by correlation with the target — and keeps only the top-ranked ones. It's fast, simple, and surprisingly effective at recovering the genuinely relevant features from among a much larger pool of noise, particularly when the true signal is concentrated in a small number of columns.
Reading an accuracy-vs-features curve
Plotting test accuracy against the number of top-ranked features kept typically shows a sharp rise through the first few features (where the real signal lives), a peak near the true number of relevant features, then a flattening or decline as increasingly irrelevant features get added back in — a direct visualization of exactly where feature selection is earning its keep.
Beyond univariate filters
Univariate ranking ignores interactions between features and can occasionally miss features that only matter in combination with others. Wrapper methods (evaluating subsets by actual downstream model performance), embedded methods (L1/Lasso regularization that zeroes out irrelevant coefficients during training itself), and mutual-information-based filters all trade simplicity for the ability to capture more complex relevance patterns than a single correlation score can.
🧪 Try it yourself: the Feature Selection Lab simulation lets you experiment with everything described above directly in your browser.