Grid Search Explained: Exhaustive Hyperparameter Tuning

Fundamentals

Designing Effective Grids

• Keep dimensions low; focus on impactful parameters
• Use log-spaced values for rates and regularization
• Combine with early stopping to save compute

Alternatives

Random search for wide spaces; Hyperband/ASHA for budget efficiency; BO for expensive evaluations.

Example

from sklearn.model_selection import GridSearchCV
GridSearchCV(model, {"C": [0.01, 0.1, 1, 10]}, cv=5).fit(X_train, y_train)

How the Algorithm Works

Algorithm

1. Define parameters and discrete value sets
2. Enumerate Cartesian product
3. Evaluate each configuration under fixed validation protocol
4. Select best by primary metric; log full results

Scheduling

• Parallelize independent trials
• Early stopping with threshold pruning
• Warm-start from adjacent settings

Examples

from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
GridSearchCV(SVC(), {"C":[0.1,1,10], "gamma":[1e-3,1e-4]}, cv=5)

Real-World Applications

Domains

• Finance: threshold tuning, calibration, class weights
• Healthcare: sensitivity-specificity trade-offs
• E-commerce: ranking and re-ranking parameters
• Recommendations: relevance-diversity balancing

Constraints

Latency budgets, fairness constraints, and explainability requirements often bound the grid scope.

Best Practices

Checklist

• Limit dimensions and use log-spaced values
• Fix seeds, splits, and record environment
• Use stratified CV and early stopping
• Cache computations; persist artifacts
• Report full grid results with uncertainty
• Document rationale and constraints

Anti-Patterns

• Massive unprincipled grids
• Mixing validation and test sets
• Ignoring interaction effects without checks
• No experiment tracking or seeds
• Over-tuning to noisy folds
• Hiding negative results

Evaluation

Validation Protocols

• Stratified k-fold for classification
• Group k-fold when leakage risk exists
• Nested CV for honest performance estimates

Uncertainty

Use confidence intervals and medians over repeats; avoid cherry-picking best fold only.

Worked Examples

SVM

GridSearchCV(SVC(), {"C":[0.01,0.1,1,10], "gamma":[1e-4,1e-3]}, cv=5)

Random Forest

GridSearchCV(RandomForestClassifier(), {"n_estimators":[200,400], "max_depth":[10,20,None]}, cv=5)

XGBoost

# Grid over eta, max_depth, subsample, colsample_bytree

Neural Networks

# Keras wrappers with grid over units and dropout

Implementation

Core Pattern

from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
pipe = Pipeline([("scaler", StandardScaler()), ("clf", SVC())])
params = {"clf__C": [0.1,1,10], "clf__gamma": [1e-3,1e-4]}
GridSearchCV(pipe, params, cv=5, n_jobs=-1).fit(X_train, y_train)

Reproducibility

• Fix seeds and record environment
• Persist splits and cache results
• Track experiments (MLflow/W&B)

The Math Behind It

Coverage and Discretization

Uniform grids yield bounded worst-case gaps along each dimension; log-spaced grids better match multiplicative sensitivities.

Curse of Dimensionality

Required points grow exponentially with dimensions; random or adaptive methods dominate in high-d.

Design of Experiments

Latin hypercube and Sobol sequences offer better space-filling properties than naive grids.

Key Parameters

Value Spacing

• Use log-spaced grids for positive scale parameters
• Keep 3–5 steps per dimension initially
• Refine around promising regions

Catalog (Examples)

• SVM C: [0.01, 0.1, 1, 10, 100]
• Gamma: [1e-4, 1e-3, 1e-2]
• RF max_depth: [None, 5, 10, 20]
• XGB eta: [0.3, 0.1, 0.03]

Training Strategy

Budgets

• Set per-trial time and epoch limits
• Use smaller budgets for coarse grids
• Increase near promising regions

Stability

• Deterministic seeds and fixed splits
• Checkpoint and resume support
• Track metrics with averages and CIs