Optimization Tools and Frameworks for ML

Fundamentals

Schedulers

• ASHA/Hyperband for budgeted allocation
• Bayesian optimization for refinement
• Evolutionary/population-based methods

Orchestrators

• Ray for distributed sweeps
• Kubernetes jobs for scale
• Airflow/Prefect for workflows

Trackers

• MLflow/W&B for runs, metrics, and artifacts
• Model registry and lineage
• Dashboards and alerts

How the Algorithm Works

ASHA/Hyperband

Budgeted promotion rules, rungs, reduction factors, and anytime performance.

Bayesian Optimization

Surrogate modeling and acquisition optimization; batch and async variants.

Evolutionary Methods

Population, mutation, crossover, and selection; PBT for online adaptation.

Best Practices

Checklist

• Define budgets and caps upfront
• Use async schedulers for scale
• Track seeds, splits, and configs
• Store artifacts and register models
• Enable approvals and audit logs

Anti-Patterns

• Unbounded sweeps without cost control
• Comparisons with different protocols
• No tracking or governance
• Ignoring failure recovery
• Leaking test sets during tuning

Worked Examples

ASHA on Ray

# Define search space, scheduler, resources, and run

BO on Ray

# Batch q-EI with parallel suggestions

Evolutionary

# Population-based training template

Implementation

Ray

# Tune with ASHA/BO and distributed trials

Kubernetes

# Jobs/Operators to scale sweeps

Tracking

• MLflow/W&B run metadata, artifacts, alerts
• Model registry integration

The Math Behind It

Bandit Regret

Successive halving relates to best-arm identification; sample complexity depends on gaps.

Convergence

BO convergence under smoothness and noise assumptions; acquisition decay.

Allocation

Rung schedules trade breadth for depth; reduction factors control promotions.