Feature Engineering: Making Easy What the Model Alone Cannot

A problem no amount of training can fix

An XOR-style label pattern — class 1 in the top-right and bottom-left quadrants of a 2D input, class 0 elsewhere — cannot be separated by any straight line, no matter how a linear classifier's weights are tuned. This isn't a training failure; it's a representational limit. The information that determines the label exists in the data, but not along either raw input axis alone.

One feature changes everything

Add the interaction feature x·y, and the exact same linear model can now separate the classes perfectly — because the sign of x·y is literally what determines the label in this pattern. The model didn't get smarter or train longer; the representation it operates on did.

Why this still matters alongside deep learning

Deep networks can, given enough data and capacity, learn interaction features like this automatically from raw inputs — part of why they've displaced manual feature engineering in vision and language, where useful representations are hard for humans to hand-design. But for tabular data, smaller datasets, and models that need to stay interpretable, hand-engineered features — ratios, differences, interaction terms, domain-specific transforms — routinely turn an unsolvable linear problem into an easy one, using a fraction of the data a network would need to learn the same representation from scratch.

Common techniques in practice

Ratios and differences between related columns, log or power transforms for skewed variables, interaction and polynomial terms, date/time decomposition (day of week, is-holiday flags), and domain-specific aggregates like rolling averages are all standard feature engineering moves — each one, in effect, doing exactly what adding x·y did in this lab: making visible to a simple model what was hidden from it in the raw representation.