Random Search Explained: Hyperparameter Tuning

Fundamentals

Search Space Design

• Use log-uniform for learning rates and regularization
• Define categorical choices with meaningful priors
• Apply constraints to avoid invalid combinations

Parallelism

Random search trivially parallelizes; use distributed runners for large-scale searches.

Templates

from sklearn.model_selection import RandomizedSearchCV
RandomizedSearchCV(model, param_distributions=space, n_iter=50, cv=3)

How the Algorithm Works

Algorithm

1. Define search space with distributions and constraints
2. Sample N candidates independently
3. Evaluate under fixed validation protocol
4. Select best and optionally refine around promising regions

Sampling Distributions

• Log-uniform for rates and regularization
• Uniform/normal for bounded continuous ranges
• Categorical with priors for algorithmic choices

Stopping Rules

Stop on budget exhaustion, plateau detection, or confidence intervals crossing.

Example

from sklearn.model_selection import RandomizedSearchCV
RandomizedSearchCV(model, space, n_iter=100, cv=3, n_jobs=-1)

Real-World Applications

Domains

• Finance: fraud, credit, risk calibration
• Healthcare: triage, imaging, prognosis
• Search/Ranking: relevance and latency trade-offs
• Recommenders: relevance-diversity balancing

KPIs and Constraints

Optimize for accuracy, latency, and fairness simultaneously with guardrail metrics.

Best Practices

Playbook

1. Define well-scaled spaces with priors
2. Run broad low-budget sweep
3. Refine around top quantile
4. Repeat with tighter ranges
5. Validate with repeats and nested CV when needed

Checklist

• Seeds and splits recorded
• Artifacts and configs versioned
• Guardrail metrics monitored
• Cost and wall-clock limits enforced
• Security and privacy checks passed

Anti-Patterns

• Flat uniform spaces for scale-sensitive params
• No repeats under heavy noise
• Untracked experiments
• Overfitting to a single validation split
• Ignoring constraints and invalid combos

Evaluation

Protocols

• Repeated CV or repeated holdout with fixed seeds
• Use medians and CIs to mitigate outliers
• Nested CV for honest selection

Worked Examples

Classification

RandomizedSearchCV(RandomForestClassifier(), space, n_iter=60, cv=5)

Regression

# Optuna sampling for XGBoost regressor

NLP

# Ray Tune sweep for transformer finetuning

Vision

# KerasTuner RandomSearch for CNN hyperparameters

Implementation

scikit-learn

from sklearn.model_selection import RandomizedSearchCV
RandomizedSearchCV(pipe, space, n_iter=80, cv=5, n_jobs=-1)

Optuna

study.optimize(objective, n_trials=200, n_jobs=8)

Ray Tune

tune.run(train_fn, config={"lr": tune.loguniform(1e-5,1e-1)})

Tracking

• Log seeds, samples, metrics, and artifacts
• Persist split indices and env lockfiles
• Export best config with metadata

The Math Behind It

Coverage Probability

Probability at least one sample lands in an ε-optimal region increases as 1 − (1 − Vε)^N where Vε is region volume.

Concentration

With i.i.d. samples, best-of-N improves sublinearly; heavy-tailed noise requires robust estimators.

Priors

Log priors align with multiplicative scales; informative priors accelerate convergence.

Key Parameters

Constraints

Respect conditional parameters (optimizer-specific options) and avoid invalid combinations via structured spaces.

Training Strategy

Budgeting

• Allocate small budgets broadly; increase for promising configs
• Use patience-based early stopping
• Cap wall-clock per trial

Stability

• Repeat trials and average metrics
• Fix seeds and maintain deterministic pipelines
• Checkpoint best and last weights