Categorical Encoding: One-Hot, Label, Frequency and Target Overfitting (2D)

Compare six ways to turn a category into numbers, and see why target encoding of rare categories looks great on training data and fails on new rows.

Switch to 3D
Controls

Presets

Parameters

Model checks

The science

The problem. A model needs numbers; a category such as a city or a product code is not one. Encoders map it to numbers: one-hot (one 0/1 column per category), label (an arbitrary integer code that invents an order), frequency (how common the category is) and target encoding (the mean of y inside the category). Target encoding is the most informative and the most dangerous, because the value it uses was computed from the very y being predicted.

Why target encoding overfits. A category with n rows has a mean that is wrong by about σ/√n even if the true effect is zero (Var(mean - β) = σ²/n). Rare categories therefore get extreme encoded values that look predictive on the training rows and are pure noise on new rows. With C unrelated categories the training R² of the C group means is (C-1)/(N-1) on average even though nothing is learnable.

Remedies shown. Smoothing shrinks a category mean toward the global mean, (n·meanc + m·global)/(n + m), so small categories move most. Out-of-fold encoding computes each training row's value from the other folds so a row never sees its own label. The page reports R² on the training rows and on an independent test set drawn from the same categories.

Simplifications. The target is a number with Gaussian noise (σ = 1) and Gaussian true effects; the model is a one-variable linear fit on the encoded value (one-hot uses all C dummies). Real pipelines use gradient boosting, cross-validation and unseen categories, which this page does not model. Checks: the null-data R² formula, the group-mean error variance, one-hot OLS = group means (Gaussian elimination), smoothing endpoints, absence of leakage for out-of-fold encoding, and R² = squared correlation.

-
-