Custom Hyperparameter Optimization Implementation

Learn how to implement custom hyperparameter optimization algorithms. Build your own optimization methods from scratch.

Introduction

Sometimes you need custom hyperparameter optimization implementations tailored to specific requirements. Building your own optimizers provides full control and enables domain-specific optimizations.

Building Custom Random Search

Basic Implementation

import random
import numpy as np

def custom_random_search(objective, search_space, n_iter=100):
    best_score = float('-inf')
    best_params = None
    results = []
    
    for i in range(n_iter):
        params = {}
        for param_name, param_range in search_space.items():
            if isinstance(param_range, list):
                params[param_name] = random.choice(param_range)
            elif isinstance(param_range, tuple):
                params[param_name] = random.uniform(param_range[0], param_range[1])
        
        score = objective(params)
        results.append((params, score))
        
        if score > best_score:
            best_score = score
            best_params = params
    
    return best_params, best_score, results
            

Custom Grid Search

Implementation

from itertools import product

def custom_grid_search(objective, search_space):
    param_names = list(search_space.keys())
    param_values = list(search_space.values())
    
    best_score = float('-inf')
    best_params = None
    
    for combination in product(*param_values):
        params = dict(zip(param_names, combination))
        score = objective(params)
        
        if score > best_score:
            best_score = score
            best_params = params
    
    return best_params, best_score
            

Custom Bayesian Optimization

Simple GP Implementation

from sklearn.gaussian_process import GaussianProcessRegressor
from scipy.optimize import minimize

class CustomBayesianOptimizer:
    def __init__(self, search_space):
        self.search_space = search_space
        self.X = []
        self.y = []
        self.gp = GaussianProcessRegressor()
    
    def acquisition_function(self, x):
        mu, sigma = self.gp.predict([x], return_std=True)
        return mu + 1.96 * sigma  # UCB
    
    def optimize(self, objective, n_iter=50):
        # Initial random samples
        for _ in range(5):
            params = self.sample_random()
            score = objective(params)
            self.X.append(params)
            self.y.append(score)
        
        # Bayesian optimization loop
        for _ in range(n_iter):
            self.gp.fit(self.X, self.y)
            next_params = self.select_next()
            score = objective(next_params)
            self.X.append(next_params)
            self.y.append(score)
        
        best_idx = np.argmax(self.y)
        return self.X[best_idx], self.y[best_idx]
            

Design Considerations

Interface Design

Consistent API
Flexible search spaces
Result tracking
Early stopping support

Modularity

Separate sampling logic
Independent acquisition functions
Pluggable surrogate models
Extensible architecture

Key Insight

Custom implementations give full control but require more work. Use when you need specific features, domain-specific optimizations, or want to learn internals. Otherwise, prefer established libraries.

Frequently Asked Questions

When should I implement custom optimization?

Implement custom optimization when you need specific features unavailable in libraries, want domain-specific optimizations, need to understand internals, or have special requirements.

How do I implement custom Random Search?

Sample random hyperparameters from search space, evaluate objective function, track best result. Use random.choice for discrete, random.uniform for continuous parameters.

How do I implement custom Grid Search?

Generate all combinations using itertools.product, evaluate each combination, track best result. Simple but computationally expensive for large spaces.

How do I implement Bayesian Optimization?

Use Gaussian Process surrogate model, implement acquisition function (UCB, EI), optimize acquisition function to select next point, update GP with new observation, repeat.

What's the advantage of custom implementation?

Full control over algorithm, can add domain-specific logic, understand internals, customize for specific needs. But requires more development and testing.