Neural Network Hyperparameter Optimization Implementation

Learn how to implement hyperparameter optimization for neural networks. Deep learning specific tuning with PyTorch and TensorFlow examples.

▶ Open the simulation

Introduction

Hyperparameter optimization for neural networks has unique considerations: long training times, GPU requirements, architecture choices, and specialized optimizers. This guide covers implementation strategies for deep learning.

Keras Tuner

Basic Usage

import kerastuner as kt

def build_model(hp):
    model = tf.keras.Sequential()
    model.add(tf.keras.layers.Dense(
        units=hp.Int('units', min_value=32, max_value=512, step=32),
        activation='relu'
    ))
    model.add(tf.keras.layers.Dropout(
        hp.Float('dropout', 0.1, 0.5)
    ))
    model.add(tf.keras.layers.Dense(10, activation='softmax'))
    
    model.compile(
        optimizer=tf.keras.optimizers.Adam(
            hp.Float('learning_rate', 1e-4, 1e-2, sampling='log')
        ),
        loss='sparse_categorical_crossentropy',
        metrics=['accuracy']
    )
    return model

tuner = kt.Hyperband(
    build_model,
    objective='val_accuracy',
    max_epochs=50,
    directory='tuning',
    project_name='my_model'
)

tuner.search(X_train, y_train, validation_split=0.2)
best_model = tuner.get_best_models()[0]
            

PyTorch with Optuna

Implementation

import torch
import torch.nn as nn
import optuna

def objective(trial):
    model = create_model(
        n_layers=trial.suggest_int('n_layers', 2, 5),
        hidden_size=trial.suggest_int('hidden_size', 64, 512),
        dropout=trial.suggest_uniform('dropout', 0.1, 0.5),
        learning_rate=trial.suggest_loguniform('lr', 1e-5, 1e-2)
    )
    
    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
    
    for epoch in range(50):
        train_epoch(model, optimizer)
        val_score = validate(model)
        
        trial.report(val_score, epoch)
        if trial.should_prune():
            raise optuna.TrialPruned()
    
    return val_score

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)
            

Architecture Search

Variable Depth Networks

def build_network(hp):
    model = Sequential()
    n_layers = hp.Int('n_layers', 2, 5)
    
    for i in range(n_layers):
        model.add(Dense(
            hp.Int(f'units_{i}', 32, 256),
            activation='relu'
        ))
        model.add(Dropout(hp.Float(f'dropout_{i}', 0.1, 0.5)))
    
    model.add(Dense(10, activation='softmax'))
    return model
            

GPU Considerations

Resource Management

Limit concurrent GPU trials
Clear GPU memory between trials
Use mixed precision training
Monitor GPU utilization

Early Stopping Integration

Keras Callbacks

callbacks = [
    tf.keras.callbacks.EarlyStopping(
        monitor='val_loss',
        patience=10,
        restore_best_weights=True
    ),
    tf.keras.callbacks.ReduceLROnPlateau(
        monitor='val_loss',
        factor=0.5,
        patience=5
    )
]

tuner.search(X_train, y_train, callbacks=callbacks)
            

Best Practice

For neural networks, use Keras Tuner for TensorFlow/Keras or Optuna/Ray Tune for PyTorch. Leverage early stopping, GPU resource management, and architecture search capabilities.

Frequently Asked Questions

How do I tune neural network hyperparameters?

Use Keras Tuner for TensorFlow/Keras or Optuna/Ray Tune for PyTorch. Tune learning rate, architecture (layers, units), dropout, batch size, optimizer settings, and training epochs.

What's Keras Tuner?

Keras Tuner is TensorFlow's hyperparameter tuning library. Provides Hyperband, Random Search, Bayesian Optimization specifically designed for Keras models with GPU support.

How do I tune PyTorch models?

Use Optuna or Ray Tune with PyTorch. Define objective function that creates model, trains it, and returns validation score. Optuna handles hyperparameter search intelligently.

How do I handle GPU memory in tuning?

Limit concurrent trials, clear GPU cache between trials (torch.cuda.empty_cache()), use batch size tuning, monitor memory usage, and consider mixed precision training.

Can I tune network architecture?

Yes, tune number of layers, layer sizes, activation functions, skip connections, etc. Use variable-depth networks or Neural Architecture Search (NAS) methods.