Hyperparameter Sensitivity Analysis: Identifying Critical Parameters

Introduction

Hyperparameter sensitivity analysis helps identify which hyperparameters have the greatest impact on model performance. This knowledge allows practitioners to prioritize tuning efforts and allocate computational resources efficiently.

What is Sensitivity Analysis?

Sensitivity analysis measures how much changes in hyperparameter values affect model performance. Highly sensitive hyperparameters cause large performance variations with small changes, while insensitive ones have minimal impact.

Why Sensitivity Analysis Matters

Prioritization

Identify which hyperparameters to tune first:

• Focus on high-sensitivity hyperparameters
• Allocate tuning time efficiently
• Avoid wasting resources on low-impact parameters

Resource Allocation

Optimize computational resources:

• More time on sensitive hyperparameters
• Less time on insensitive ones
• Better overall results with same resources

Understanding Model Behavior

Gain insights into model dynamics:

• Which hyperparameters control which aspects
• How hyperparameters interact
• Model robustness to hyperparameter changes

Methods for Sensitivity Analysis

One-at-a-Time Analysis

Vary one hyperparameter while keeping others fixed:

• Advantages: Simple, interpretable
• Disadvantages: Misses interactions
• Use Case: Initial exploration

Global Sensitivity Analysis

Examine sensitivity across entire parameter space:

• Advantages: Captures interactions
• Disadvantages: Computationally expensive
• Use Case: Deep understanding

Variance-Based Methods

Measure variance in performance due to hyperparameter changes:

• Sobol indices
• Variance decomposition
• Quantify contribution of each hyperparameter

Common Sensitivity Patterns

Highly Sensitive Hyperparameters

These typically have large impact:

Moderately Sensitive Hyperparameters

These have moderate impact:

• Batch size (neural networks)
• Regularization strength
• Optimizer choice
• Activation functions

Low Sensitivity Hyperparameters

These have minimal impact:

• Random seed (affects reproducibility, not performance)
• Some preprocessing parameters
• Logging and checkpointing settings

Conducting Sensitivity Analysis

Step 1: Define Baseline

Establish baseline performance with default hyperparameters:

• Use library defaults
• Or literature values
• Document baseline performance

Step 2: Vary Hyperparameters

Systematically vary each hyperparameter:

• Use grid or random search
• Track performance changes
• Measure sensitivity metrics

Step 3: Measure Impact

Quantify sensitivity:

• Performance variance
• Range of performance changes
• Statistical significance

Step 4: Rank by Importance

Prioritize hyperparameters:

• Rank by sensitivity
• Identify critical ones
• Plan tuning strategy

Interpreting Sensitivity Results

High Sensitivity

Indicates:

• Hyperparameter has large impact
• Worth spending more time tuning
• Requires careful selection

Low Sensitivity

Indicates:

• Hyperparameter has minimal impact
• Default values may be sufficient
• Less critical to tune

Non-Linear Sensitivity

Some hyperparameters show non-linear effects:

• Optimal values exist
• Performance degrades away from optimal
• Requires careful exploration

Hyperparameter Interactions

Identifying Interactions

Some hyperparameters interact:

• Learning rate × batch size
• Depth × regularization
• Model size × dropout

Analyzing Interactions

Consider:

• Joint sensitivity
• Conditional effects
• Optimal combinations

Practical Recommendations

Start with High-Sensitivity Hyperparameters

• Learning rate for neural networks
• Max depth for tree-based models
• C parameter for SVM

Use Sensitivity to Guide Search

• Wider ranges for sensitive hyperparameters
• Finer resolution for sensitive ones
• Less exploration for insensitive ones

Consider Computational Costs

• Balance sensitivity with tuning cost
• Some sensitive hyperparameters are expensive to tune
• Prioritize high-sensitivity, low-cost ones