Categorical Encoding: One-Hot, Label, Frequency and Target Overfitting

One pillar per category: blue marker = raw target mean, gold ring = true effect, green = smoothed; the rods are the estimation error. Drag to orbit.

Switch to 2D
Controls

Presets

Parameters

Quality

Camera

Model checks

The science

The problem. A model needs numbers; a category such as a city or a product code is not one. Encoders map it to numbers: one-hot (one 0/1 column per category), label (an arbitrary integer code that invents an order), frequency (how common the category is) and target encoding (the mean of y inside the category). Target encoding is the most informative and the most dangerous, because the value it uses was computed from the very y being predicted.

Why target encoding overfits. A category with n rows has a mean that is wrong by about σ/√n even if the true effect is zero (Var(mean - β) = σ²/n). Rare categories therefore get extreme encoded values that look predictive on the training rows and are pure noise on new rows. With C unrelated categories the training R² of the C group means is (C-1)/(N-1) on average even though nothing is learnable.

Remedies shown. Smoothing shrinks a category mean toward the global mean, (n·meanc + m·global)/(n + m), so small categories move most. Out-of-fold encoding computes each training row's value from the other folds so a row never sees its own label. The page reports R² on the training rows and on an independent test set drawn from the same categories.

Simplifications. The target is a number with Gaussian noise (σ = 1) and Gaussian true effects; the model is a one-variable linear fit on the encoded value (one-hot uses all C dummies). Real pipelines use gradient boosting, cross-validation and unseen categories, which this page does not model. Checks: the null-data R² formula, the group-mean error variance, one-hot OLS = group means (Gaussian elimination), smoothing endpoints, absence of leakage for out-of-fold encoding, and R² = squared correlation.

Loading model...
Loading Blender model...
-
-
Made with 🟠 Blender · Three.js