Hyperparameter Optimization Code Best Practices

Learn best practices for implementing hyperparameter optimization code. Code organization, error handling, logging, and production considerations.

▶ Open the simulation

Introduction

Writing production-ready hyperparameter optimization code requires attention to code organization, error handling, logging, reproducibility, and maintainability. Following best practices ensures robust and reliable implementations.

Code Organization

Modular Structure

# Separate files for organization
# config.py - Hyperparameter spaces
# objective.py - Objective functions
# optimize.py - Optimization logic
# utils.py - Helper functions
            

Configuration Management

# Use YAML or JSON for configs
import yaml

with open('config.yaml') as f:
    config = yaml.safe_load(f)

param_grid = config['hyperparameters']
            

Error Handling

Robust Objective Functions

def objective(trial):
    try:
        params = suggest_parameters(trial)
        model = create_model(params)
        score = evaluate(model)
        return score
    except Exception as e:
        logger.error(f"Trial failed: {e}")
        return float('-inf')  # or raise optuna.TrialPruned()
            

Validation

Validate hyperparameter ranges
Check data availability
Verify model compatibility
Handle edge cases

Logging

Comprehensive Logging

import logging

logging.basicConfig(
    filename='optimization.log',
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s'
)

def objective(trial):
    params = suggest_parameters(trial)
    logging.info(f"Trial {trial.number}: {params}")
    
    score = evaluate(model)
    logging.info(f"Trial {trial.number}: score={score}")
    
    return score
            

Reproducibility

Setting Seeds

import random
import numpy as np
import torch

def set_seed(seed):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    if torch.cuda.is_available():
        torch.cuda.manual_seed_all(seed)

set_seed(42)
            

Version Control

Commit code regularly
Version datasets
Document dependencies
Track experiments

Performance Optimization

Caching

from functools import lru_cache

@lru_cache(maxsize=1000)
def evaluate_model(params_tuple):
    # Expensive evaluation
    return score
            

Parallelization

Use n_jobs=-1 when possible
Profile bottleneck operations
Optimize data loading
Batch operations

Best Practice

Write clean, modular, well-documented code with proper error handling and logging. Always set seeds for reproducibility and validate results thoroughly before deployment.

Testing

Unit Tests

import unittest

class TestHyperparameterOptimization(unittest.TestCase):
    def test_objective_function(self):
        trial = create_mock_trial()
        score = objective(trial)
        self.assertIsInstance(score, float)
            

Documentation

Code Comments

Document hyperparameter choices
Explain algorithm selection
Note known limitations
Include usage examples

Frequently Asked Questions

How should I organize hyperparameter optimization code?

Organize into modules: config for hyperparameter spaces, objective for evaluation functions, optimize for main logic, utils for helpers. Use configuration files for flexibility.

How do I handle errors in optimization?

Wrap objective functions in try-except blocks, log errors, return default values or raise TrialPruned. Validate inputs and handle edge cases gracefully.

What should I log during optimization?

Log hyperparameters, scores, errors, timing, resource usage, and progress. Use structured logging for easy analysis. Include timestamps and trial identifiers.

How do I ensure reproducibility?

Set random seeds for all libraries (random, numpy, tensorflow, pytorch), use fixed random_state parameters, document versions, and commit code.

Can I cache expensive evaluations?

Yes, use functools.lru_cache for deterministic functions, or implement custom caching with hashable parameter representations. Saves computation for repeated configurations.