What a single split actually measures
A train/test split answers a narrow question: how did the model perform on this particular random subset held out as a test set? With a small or noisy dataset, which specific points land in that subset can swing the measured error substantially — the model hasn't changed at all, only the sample used to evaluate it has.
K-fold cross-validation: average over many splits
K-fold cross-validation partitions the data into k folds, trains on k-1 of them and validates on the remaining one, repeating k times so every point gets used for validation exactly once — then averages the results. This trades k times the compute for a far more stable estimate, plus a standard deviation across folds that tells you how much that average could still vary.
Choosing k
Larger k means each fold's training set is closer in size to the full dataset, reducing bias in the estimate, but increases compute cost and shrinks each validation fold. k=5 or k=10 are the standard defaults. The extreme case, leave-one-out cross-validation (k equal to the sample size), has the lowest possible bias but is the most expensive and can have surprisingly high variance for some models.
Using CV for model selection, not just evaluation
Comparing CV mean error across candidate hyperparameters — polynomial degree, regularization strength, tree depth — is exactly what cross-validation is designed for. Picking based on a single split risks selecting whichever setting happened to get lucky on that particular held-out 20%, which is a subtly different (and worse) mistake than just reporting an unreliable final number.
🧪 Try it yourself: the Cross-Validation Lab simulation lets you experiment with everything described above directly in your browser.