Implementing Hyperparameter Optimization in Python: Complete Guide

Learn how to implement hyperparameter optimization in Python. Step-by-step guide to implementing Grid Search, Random Search, and Bayesian Optimization.

▶ Open the simulation

Introduction

Implementing hyperparameter optimization requires understanding libraries, APIs, and best practices. This guide provides practical Python implementations for common hyperparameter optimization methods.

Using Scikit-Learn

GridSearchCV

Exhaustive grid search with cross-validation:

from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC

param_grid = {
    'C': [0.1, 1, 10],
    'kernel': ['linear', 'rbf'],
    'gamma': ['scale', 'auto']
}

grid_search = GridSearchCV(
    SVC(),
    param_grid,
    cv=5,
    scoring='accuracy',
    n_jobs=-1
)

grid_search.fit(X_train, y_train)
best_params = grid_search.best_params_
            

RandomizedSearchCV

Random search with cross-validation:

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import uniform, loguniform

param_distributions = {
    'C': loguniform(1e-3, 1e2),
    'kernel': ['linear', 'rbf', 'poly'],
    'gamma': ['scale', 'auto']
}

random_search = RandomizedSearchCV(
    SVC(),
    param_distributions,
    n_iter=100,
    cv=5,
    scoring='accuracy',
    n_jobs=-1,
    random_state=42
)

random_search.fit(X_train, y_train)
            

Using Optuna

Basic Setup

import optuna

def objective(trial):
    C = trial.suggest_loguniform('C', 1e-3, 1e2)
    kernel = trial.suggest_categorical('kernel', ['linear', 'rbf'])
    gamma = trial.suggest_categorical('gamma', ['scale', 'auto'])
    
    model = SVC(C=C, kernel=kernel, gamma=gamma)
    score = cross_val_score(model, X_train, y_train, cv=5).mean()
    return score

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)
best_params = study.best_params
            

Advanced Features

Pruning with MedianPruner
Multi-objective optimization
Visualization
Distributed optimization

Using Ray Tune

Basic Example

from ray import tune
from ray.tune.schedulers import ASHAScheduler

def train_model(config):
    C = config['C']
    kernel = config['kernel']
    
    model = SVC(C=C, kernel=kernel)
    score = cross_val_score(model, X_train, y_train, cv=5).mean()
    tune.report(accuracy=score)

analysis = tune.run(
    train_model,
    config={
        'C': tune.loguniform(1e-3, 1e2),
        'kernel': tune.choice(['linear', 'rbf'])
    },
    num_samples=100,
    scheduler=ASHAScheduler()
)
            

Best Practices

Code Organization

Separate objective function
Modular hyperparameter space
Result logging
Reproducibility (seeds)

Performance Optimization

Parallel evaluation
Caching results
Early stopping
Resource management

Key Insight

Use established libraries (scikit-learn, Optuna, Ray Tune) rather than implementing from scratch. They provide tested, optimized implementations with useful features.

Implementation Checklist

Define hyperparameter search space
Set up cross-validation
Choose optimization algorithm
Implement evaluation function
Run optimization
Analyze results
Validate best hyperparameters

Frequently Asked Questions

How do I implement Grid Search in Python?

Use sklearn.model_selection.GridSearchCV. Define param_grid dictionary, create GridSearchCV object with estimator and parameters, call fit(), and access best_params_ attribute.

What's the difference between GridSearchCV and RandomizedSearchCV?

GridSearchCV evaluates all combinations exhaustively. RandomizedSearchCV samples random combinations. Use GridSearchCV for small spaces, RandomizedSearchCV for large spaces.

How do I use Optuna for hyperparameter optimization?

Define objective function that takes trial, use trial.suggest_* methods to sample hyperparameters, create study, call optimize(), and access best_params. Optuna handles search intelligently.

Can I parallelize hyperparameter optimization?

Yes, use n_jobs=-1 in scikit-learn, or use Ray Tune for distributed optimization. GridSearchCV and RandomizedSearchCV support parallel evaluation natively.

How do I log optimization results?

Save results from GridSearchCV.cv_results_, use Optuna's built-in logging, or implement custom logging. Most frameworks provide result tracking and visualization.