Feature Engineering Fundamentals

Feature engineering is where most ML value is created. A simple model with great features beats a complex model with poor features. This is the work that actually moves metrics - transforming raw data into signals that models can learn from.

Where It Is Used

Every ML system relies on feature engineering:

• Fraud detection: Velocity features, behavioral patterns, network signals
• Recommendations: User preferences, item popularity, interaction history
• Credit scoring: Payment history aggregations, debt ratios, stability indicators
• Demand forecasting: Seasonality, trends, promotional effects
• Churn prediction: Engagement trends, support interactions, usage patterns

The best ML teams spend 60-80% of their time on features, not models.

Typical Data You Will See

Raw data before feature engineering:

Feature engineering transforms this into model-ready format: - Numeric features scaled appropriately - Categorical features encoded - Missing values handled - Temporal patterns extracted - Interactions captured

What You Do in Practice

1. Understand the business problem - What signal would help predict the target?
2. Explore the raw data - What is available, what is the quality, what is missing?
3. Define the prediction point - What information is available at prediction time?
4. Extract base features - Direct mappings from raw data
5. Create aggregations - Sums, counts, averages over time windows
6. Build ratio features - Current vs historical, entity vs population
7. Encode categoricals - Choose encoding based on cardinality and model
8. Handle missing values - Imputation or indicator features
9. Validate feature quality - Check distributions, correlations, leakage
10. Test in production - Ensure features compute the same way in serving

Baseline First

Start with the simplest feature set that captures the problem:

1. Direct mappings: Numeric columns as-is, one-hot encoded categoricals
2. Basic aggregations: Count, sum, mean of key events
3. Recency features: Days since last event
4. Simple ratios: This period vs last period

This baseline: - Establishes performance floor quickly - Identifies data quality issues early - Provides interpretable starting point - Enables fast iteration

Add complexity only when baseline performance plateaus.

Common Patterns and When to Choose

Numeric Transformations

Categorical Encoding

Temporal Features

Interaction Features

Feature Engineering Patterns

Aggregation framework:

For each entity (user, product, session):
  For each event type (purchase, click, view):
    For each time window (1d, 7d, 30d, 90d):
      Compute: count, sum, mean, max, min, std

Behavioral signals: - Consistency: std of activity across time periods - Trend: comparison of recent vs historical behavior - Diversity: count of unique categories interacted with - Intensity: activity rate per time unit

Text features: - Length: character count, word count - Patterns: regex matches, keyword presence - Embeddings: TF-IDF, sentence transformers

Missing value strategies: - Impute with median/mode for random missingness - Impute with domain-specific value (e.g., 0 for count features) - Create binary indicator: is_feature_missing - Leave as-is for tree models that handle missing values

Evaluation and Metrics

Feature quality metrics:

Pitfalls: - Feature leakage: Using information from after prediction time - Target leakage: Feature that encodes the label directly or indirectly - Distribution shift: Feature computed differently in training vs serving - High cardinality surprises: New categories in production

Failure Modes and Debugging Checklist

• [ ] Leakage: Feature has impossibly high correlation with target
• [ ] Training-serving skew: Different code paths for training and inference
• [ ] Stale features: Online features not updated, serving outdated values
• [ ] NULL explosion: Upstream data issue causes mass missing values
• [ ] Type mismatch: Integer overflow, string encoding issues
• [ ] Time zone bugs: Aggregation windows computed in wrong timezone
• [ ] Off-by-one errors: Time windows include future data
• [ ] Join fanout: One-to-many join inflates feature values

Production Checklist

Feature development: - [ ] Feature definition documented - [ ] Unit tests for feature logic - [ ] Tested on holdout data - [ ] Checked for leakage - [ ] Compute cost acceptable

Feature deployment: - [ ] Same logic in training and serving - [ ] Monitoring for feature drift - [ ] Alerting for NULL rates - [ ] Backfill strategy defined - [ ] Versioning in place

Feature maintenance: - [ ] Regular quality audits - [ ] Deprecation process for unused features - [ ] Documentation updated - [ ] Dependencies tracked

Tools and Stack

Example Mini-Case

Scenario: Improve churn prediction model for subscription service.

Current state: Model uses 12 features (demographics, plan type, tenure). AUC 0.68.

Feature engineering sprint: 1. Engagement features: Sessions last 7/30/90 days, feature usage counts 2. Trend features: Sessions this week vs 4-week average 3. Support signals: Tickets opened, resolution time, sentiment 4. Payment features: Failed payments, payment method changes 5. Recency features: Days since last login, last feature use

Result: 45 new features. After selection, 23 features in final model. AUC improved to 0.79.

Key insight: Trend features (engagement declining) were most predictive, not static features.

Related Pages

• Index: ML Field Guide
• Binary Classification in Production
• Feature Stores in Practice
• Data Quality Monitoring

Further Reading

• Feature Engineering for Machine Learning - Alice Zheng
• scikit-learn Preprocessing Guide
• Feast Documentation
• Great Expectations Documentation

Try it live

Everything above runs in your browser — open Feature Engineering: Building Predictive Models and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.