Designing Hyperparameter Search Spaces: Complete Guide

Master hyperparameter search space design. Learn how to define effective search spaces for hyperparameter optimization algorithms.

▶ Open the simulation

Introduction

Designing effective hyperparameter search spaces is crucial for successful hyperparameter optimization. A well-designed search space balances exploration of promising regions with computational efficiency.

What is a Search Space?

A search space defines the set of all possible hyperparameter combinations that can be explored during optimization. It includes:

The hyperparameters to tune
Valid ranges or sets of values for each
Constraints and relationships between hyperparameters

Types of Search Spaces

Continuous Search Spaces

For continuous hyperparameters that can take any value in a range:

Unbounded: All real numbers (rarely used)
Bounded: Values within specified min and max
Log Scale: For hyperparameters that span orders of magnitude

# Example: Continuous search space
learning_rate = [0.0001, 0.1]  # Linear scale
learning_rate_log = [1e-4, 1e-1]  # Log scale (better for learning rates)
            

Discrete Search Spaces

For discrete hyperparameters with specific values:

Finite Set: Explicit list of values
Integer Range: Integer values within range
Categorical: Non-numeric categories

# Example: Discrete search space
batch_size = [16, 32, 64, 128, 256]
num_layers = range(2, 10)
optimizer = ['adam', 'sgd', 'rmsprop']
            

Mixed Search Spaces

Most real-world problems involve mixed spaces:

Continuous and discrete hyperparameters
Different scales for different hyperparameters
Conditional hyperparameters

Key Principles

1. Start Broad, Then Narrow

Begin with wide ranges based on literature and domain knowledge:

Initial exploration covers broad possibilities
Identify promising regions
Narrow focus to high-performing areas
Increase resolution in promising regions

2. Use Appropriate Scales

Choose scales that match hyperparameter characteristics:

Linear Scale: For hyperparameters with uniform impact
Log Scale: For hyperparameters spanning orders of magnitude
Custom Scales: For domain-specific cases

3. Consider Interactions

Account for hyperparameter interactions:

Some combinations may be invalid
Certain values depend on others
Joint optimization considers interactions

Common Search Space Patterns

Neural Networks

search_space = {
    'learning_rate': [1e-5, 1e-1],  # Log scale
    'batch_size': [16, 32, 64, 128, 256],  # Discrete
    'num_layers': range(2, 8),  # Integer range
    'num_neurons': range(32, 512),  # Integer range
    'dropout_rate': [0.1, 0.5],  # Continuous
    'optimizer': ['adam', 'sgd', 'rmsprop']  # Categorical
}
            

Tree-Based Models

search_space = {
    'max_depth': range(3, 20),
    'min_samples_split': range(2, 20),
    'min_samples_leaf': range(1, 10),
    'n_estimators': range(50, 500),
    'max_features': ['sqrt', 'log2', None]
}
            

Support Vector Machines

search_space = {
    'C': [0.1, 100],  # Log scale
    'gamma': ['scale', 'auto', 0.001, 10],  # Mixed
    'kernel': ['linear', 'rbf', 'poly']
}
            

Log Scale vs Linear Scale

When to Use Log Scale

Use logarithmic scale for hyperparameters that span orders of magnitude:

Learning Rate: Often ranges from 0.0001 to 0.1
Regularization: C parameter in SVM (0.1 to 100)
Batch Size: Sometimes varies widely

When to Use Linear Scale

Use linear scale for hyperparameters with uniform impact:

Dropout Rate: 0.0 to 1.0
Number of Layers: Small integer range
Most architectural parameters

Key Insight

Many hyperparameters should be searched on log scale because their impact often scales logarithmically. Learning rate is a classic example—the difference between 0.001 and 0.01 is more significant than between 0.1 and 0.11.

Search Space Size Considerations

Computational Complexity

The search space size grows exponentially with the number of hyperparameters:

2 hyperparameters: 100 combinations
5 hyperparameters: 100,000 combinations
10 hyperparameters: 10^10 combinations

Strategies for Large Spaces

Prioritize: Focus on high-impact hyperparameters
Stage Tuning: Tune in stages, not all at once
Reduce Ranges: Narrow ranges based on initial results
Efficient Methods: Use Bayesian optimization instead of grid search

Conditional Search Spaces

Hyperparameter Dependencies

Some hyperparameters depend on others:

Architecture-Dependent: Number of neurons per layer depends on number of layers
Algorithm-Dependent: Kernel-specific parameters depend on kernel choice
Problem-Dependent: Some hyperparameters only relevant for certain problems

Handling Conditionals

Define conditional search spaces
Use hierarchical search strategies
Consider automated tools that handle conditionals

Search Space Validation

Checking Feasibility

Ensure search space is well-defined:

All ranges are valid
No contradictions or invalid combinations
Ranges match hyperparameter types
Constraints are properly specified

Testing Boundaries

Test boundary values:

Minimum values
Maximum values
Edge cases
Extreme combinations

Best Practices

Do's

Start with literature-based ranges
Use log scale for appropriate hyperparameters
Consider computational constraints
Document your search space decisions
Start broad, then narrow
Consider hyperparameter interactions

Don'ts

Don't make search space too large
Don't ignore scale considerations
Don't forget about computational costs
Don't tune too many hyperparameters at once
Don't ignore domain knowledge

Frequently Asked Questions

What is a hyperparameter search space?

A search space defines all possible hyperparameter combinations that can be explored during optimization. It includes valid ranges or sets of values for each hyperparameter and any constraints or relationships between them.

Should I use log scale or linear scale for hyperparameters?

Use log scale for hyperparameters that span orders of magnitude (like learning rate, regularization strength). Use linear scale for hyperparameters with uniform impact across their range (like dropout rate, number of layers).

How large should my search space be?

Start with a reasonably broad search space based on literature and domain knowledge. Balance exploration with computational resources. As you find promising regions, narrow the search space and increase resolution in those areas.

Can I search all hyperparameters at once?

Technically yes, but it's often better to prioritize high-impact hyperparameters first. Searching too many at once exponentially increases the search space size and computational requirements. Consider staged tuning.

How do I handle conditional hyperparameters?

Define conditional search spaces where some hyperparameters depend on others. Use hierarchical search strategies or automated tools that can handle conditionals. For example, layer-specific parameters only apply when that layer exists.