Optuna Implementation Guide: Bayesian Hyperparameter Optimization

Learn how to implement Optuna for hyperparameter optimization. Complete guide with examples for Bayesian optimization, pruning, and advanced features.

Introduction

Optuna is a powerful hyperparameter optimization framework that implements Bayesian optimization and other advanced algorithms. It provides an intuitive API and powerful features for efficient hyperparameter tuning.

Basic Optuna Setup

Simple Example

import optuna
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

def objective(trial):
    n_estimators = trial.suggest_int('n_estimators', 50, 300)
    max_depth = trial.suggest_int('max_depth', 3, 20)
    min_samples_split = trial.suggest_int('min_samples_split', 2, 20)
    
    model = RandomForestClassifier(
        n_estimators=n_estimators,
        max_depth=max_depth,
        min_samples_split=min_samples_split,
        random_state=42
    )
    
    score = cross_val_score(model, X_train, y_train, cv=5).mean()
    return score

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)
print(f"Best params: {study.best_params}")
print(f"Best score: {study.best_value}")
            

Sampling Methods

Log-Uniform Sampling

learning_rate = trial.suggest_loguniform('learning_rate', 1e-5, 1e-1)

Categorical Sampling

kernel = trial.suggest_categorical('kernel', ['linear', 'rbf', 'poly'])

Uniform Sampling

dropout_rate = trial.suggest_uniform('dropout_rate', 0.1, 0.5)

Pruning

Median Pruner

import optuna.pruners

study = optuna.create_study(
    direction='maximize',
    pruner=optuna.pruners.MedianPruner()
)

def objective(trial):
    # ... model setup ...
    for epoch in range(100):
        # ... training ...
        score = validate(model, X_val, y_val)
        
        trial.report(score, epoch)
        if trial.should_prune():
            raise optuna.TrialPruned()
    
    return final_score
            

Advanced Features

Multi-Objective Optimization

def objective(trial):
    # ... model setup ...
    accuracy = evaluate_accuracy(model)
    model_size = get_model_size(model)
    return accuracy, model_size

study = optuna.create_study(
    directions=['maximize', 'minimize']
)
study.optimize(objective, n_trials=100)
            

Conditional Search Spaces

def objective(trial):
    model_type = trial.suggest_categorical('model_type', ['svm', 'rf'])
    
    if model_type == 'svm':
        C = trial.suggest_loguniform('C', 1e-3, 1e2)
        kernel = trial.suggest_categorical('kernel', ['linear', 'rbf'])
    else:
        n_estimators = trial.suggest_int('n_estimators', 50, 300)
        max_depth = trial.suggest_int('max_depth', 3, 20)
    
    # ... rest of objective ...
            

Visualization

Plotting Functions

import optuna.visualization as vis

# Optimization history
vis.plot_optimization_history(study)

# Parameter importance
vis.plot_param_importances(study)

# Parallel coordinate plot
vis.plot_parallel_coordinate(study)
            

Study Persistence

SQLite Storage

study = optuna.create_study(
    study_name='my_study',
    storage='sqlite:///db.sqlite3',
    load_if_exists=True,
    direction='maximize'
)
            

Best Practice

Use Optuna for advanced hyperparameter optimization when you need Bayesian methods, pruning, multi-objective optimization, or distributed tuning. It's more powerful than scikit-learn but requires more setup.

Frequently Asked Questions

How do I install Optuna?

Install with pip: pip install optuna. For visualization: pip install optuna[visualization]. For distributed: pip install optuna[integration].

What's the basic Optuna workflow?

Define objective function taking trial, use trial.suggest_* to sample hyperparameters, create study with direction, call optimize(), access best_params and best_value.

How do I use pruning in Optuna?

Create study with pruner (e.g., MedianPruner), call trial.report(score, step) during training, check trial.should_prune() and raise TrialPruned() if needed.

Can I resume Optuna studies?

Yes, use storage backend (SQLite, PostgreSQL, etc.) and load_if_exists=True. Studies persist automatically with storage backend.

How do I visualize Optuna results?

Use optuna.visualization functions: plot_optimization_history(), plot_param_importances(), plot_parallel_coordinate(). Requires plotly.