Scikit-Learn Hyperparameter Optimization Implementation

Learn scikit-learn implementation of hyperparameter optimization. Complete guide to GridSearchCV, RandomizedSearchCV, and advanced techniques.

▶ Open the simulation

Introduction

Scikit-learn provides powerful tools for hyperparameter optimization through GridSearchCV and RandomizedSearchCV. These classes combine hyperparameter search with cross-validation, making implementation straightforward.

GridSearchCV Deep Dive

Complete Example

from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=1000, n_features=20)

param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [3, 5, 7, None],
    'min_samples_split': [2, 5, 10],
    'max_features': ['sqrt', 'log2']
}

rf = RandomForestClassifier(random_state=42)

grid_search = GridSearchCV(
    estimator=rf,
    param_grid=param_grid,
    cv=5,
    scoring='accuracy',
    n_jobs=-1,
    verbose=1,
    return_train_score=True
)

grid_search.fit(X, y)

print(f"Best parameters: {grid_search.best_params_}")
print(f"Best score: {grid_search.best_score_}")
            

Key Parameters

estimator: Model to tune
param_grid: Dictionary of hyperparameters
cv: Cross-validation strategy
scoring: Evaluation metric
n_jobs: Parallel jobs (-1 for all cores)
verbose: Progress output

RandomizedSearchCV

With Distributions

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint, uniform, loguniform

param_distributions = {
    'n_estimators': randint(50, 300),
    'max_depth': randint(3, 20),
    'min_samples_split': randint(2, 20),
    'C': loguniform(1e-3, 1e2)
}

random_search = RandomizedSearchCV(
    estimator=rf,
    param_distributions=param_distributions,
    n_iter=100,
    cv=5,
    scoring='accuracy',
    n_jobs=-1,
    random_state=42
)

random_search.fit(X, y)
            

Accessing Results

Best Hyperparameters

best_params = grid_search.best_params_
best_score = grid_search.best_score_
best_model = grid_search.best_estimator_
            

All Results

results = grid_search.cv_results_
for mean_score, params in zip(results['mean_test_score'], results['params']):
    print(f"{mean_score:.3f} with {params}")
            

Advanced Features

Custom Scoring

from sklearn.metrics import make_scorer, f1_score

f1_scorer = make_scorer(f1_score, average='weighted')
grid_search = GridSearchCV(
    estimator=rf,
    param_grid=param_grid,
    scoring=f1_scorer,
    cv=5
)
            

Multiple Metrics

grid_search = GridSearchCV(
    estimator=rf,
    param_grid=param_grid,
    scoring=['accuracy', 'f1', 'roc_auc'],
    refit='accuracy',
    cv=5
)
            

Best Practices

Data Preparation

Train-test split before GridSearchCV
Use cross-validation properly
Scale features if needed
Handle missing values

Parameter Grid Design

Start with wide ranges
Use log scale for appropriate parameters
Consider computational cost
Document choices

Pro Tip

Always split data into train/test before GridSearchCV. GridSearchCV does internal cross-validation, but you need separate test set for final evaluation.

Pipeline Integration

With Preprocessing

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('classifier', RandomForestClassifier())
])

param_grid = {
    'classifier__n_estimators': [50, 100],
    'classifier__max_depth': [3, 5]
}

grid_search = GridSearchCV(pipeline, param_grid, cv=5)
            

Frequently Asked Questions

How do I use GridSearchCV?

Create GridSearchCV object with estimator, param_grid, cv, and scoring. Call fit() with training data. Access best_params_, best_score_, and best_estimator_ attributes.

What's the difference between GridSearchCV and RandomizedSearchCV?

GridSearchCV evaluates all parameter combinations exhaustively. RandomizedSearchCV samples random combinations. Use GridSearchCV for small spaces, RandomizedSearchCV for large.

How do I parallelize GridSearchCV?

Set n_jobs=-1 to use all CPU cores, or specify number of jobs. GridSearchCV automatically parallelizes cross-validation folds and parameter combinations.

How do I access all results from GridSearchCV?

Use cv_results_ attribute which contains mean_test_score, std_test_score, params, and other metrics for all parameter combinations evaluated.

Can I use GridSearchCV with pipelines?

Yes, GridSearchCV works with Pipeline objects. Use double underscore notation: 'step__parameter' to specify parameters for pipeline steps.