Implementing Hyperparameter Optimization in Python: Complete Guide

Learn how to implement hyperparameter optimization in Python. Step-by-step guide to implementing Grid Search, Random Search, and Bayesian Optimization.

Introduction

Implementing hyperparameter optimization requires understanding libraries, APIs, and best practices. This guide provides practical Python implementations for common hyperparameter optimization methods.

Using Scikit-Learn

GridSearchCV

Exhaustive grid search with cross-validation:

from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC

param_grid = {
    'C': [0.1, 1, 10],
    'kernel': ['linear', 'rbf'],
    'gamma': ['scale', 'auto']
}

grid_search = GridSearchCV(
    SVC(),
    param_grid,
    cv=5,
    scoring='accuracy',
    n_jobs=-1
)

grid_search.fit(X_train, y_train)
best_params = grid_search.best_params_
            

RandomizedSearchCV

Random search with cross-validation:

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import uniform, loguniform

param_distributions = {
    'C': loguniform(1e-3, 1e2),
    'kernel': ['linear', 'rbf', 'poly'],
    'gamma': ['scale', 'auto']
}

random_search = RandomizedSearchCV(
    SVC(),
    param_distributions,
    n_iter=100,
    cv=5,
    scoring='accuracy',
    n_jobs=-1,
    random_state=42
)

random_search.fit(X_train, y_train)
            

Using Optuna

Basic Setup

import optuna

def objective(trial):
    C = trial.suggest_loguniform('C', 1e-3, 1e2)
    kernel = trial.suggest_categorical('kernel', ['linear', 'rbf'])
    gamma = trial.suggest_categorical('gamma', ['scale', 'auto'])
    
    model = SVC(C=C, kernel=kernel, gamma=gamma)
    score = cross_val_score(model, X_train, y_train, cv=5).mean()
    return score

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)
best_params = study.best_params
            

Advanced Features

Pruning with MedianPruner
Multi-objective optimization
Visualization
Distributed optimization

Using Ray Tune

Basic Example

from ray import tune
from ray.tune.schedulers import ASHAScheduler

def train_model(config):
    C = config['C']
    kernel = config['kernel']
    
    model = SVC(C=C, kernel=kernel)
    score = cross_val_score(model, X_train, y_train, cv=5).mean()
    tune.report(accuracy=score)

analysis = tune.run(
    train_model,
    config={
        'C': tune.loguniform(1e-3, 1e2),
        'kernel': tune.choice(['linear', 'rbf'])
    },
    num_samples=100,
    scheduler=ASHAScheduler()
)
            

Best Practices

Code Organization

Separate objective function
Modular hyperparameter space
Result logging
Reproducibility (seeds)

Performance Optimization

Parallel evaluation
Caching results
Early stopping
Resource management

Key Insight

Use established libraries (scikit-learn, Optuna, Ray Tune) rather than implementing from scratch. They provide tested, optimized implementations with useful features.

Implementation Checklist

Define hyperparameter search space
Set up cross-validation
Choose optimization algorithm
Implement evaluation function
Run optimization
Analyze results
Validate best hyperparameters

Frequently Asked Questions

How do I implement Grid Search in Python?

Use sklearn.model_selection.GridSearchCV. Define param_grid dictionary, create GridSearchCV object with estimator and parameters, call fit(), and access best_params_ attribute.

What's the difference between GridSearchCV and RandomizedSearchCV?

GridSearchCV evaluates all combinations exhaustively. RandomizedSearchCV samples random combinations. Use GridSearchCV for small spaces, RandomizedSearchCV for large spaces.

How do I use Optuna for hyperparameter optimization?

Define objective function that takes trial, use trial.suggest_* methods to sample hyperparameters, create study, call optimize(), and access best_params. Optuna handles search intelligently.

Can I parallelize hyperparameter optimization?

Yes, use n_jobs=-1 in scikit-learn, or use Ray Tune for distributed optimization. GridSearchCV and RandomizedSearchCV support parallel evaluation natively.

How do I log optimization results?

Save results from GridSearchCV.cv_results_, use Optuna's built-in logging, or implement custom logging. Most frameworks provide result tracking and visualization.