Each glowing dot is a candidate neural-network architecture,
placed at coordinates (depth, width) over a fitness landscape
of validation accuracy — the peaks are architecture families
that generalize well, the valleys are poor designs. This is
exactly the search HPO tools like Optuna and AutoKeras perform,
just with many more hyperparameters than the two shown here.
acc(x,z) = Σ Aᵢ · exp(−dᵢ² / 2σᵢ²) (fitness landscape)
child = elite + N(0, mutation²) (Gaussian mutation)
next-gen = top-k elite ∪ mutate(elite) (evolutionary selection)
- Population size — how many architectures are trained and evaluated per generation.
- Mutation strength — how far a child's hyperparameters drift from its elite parent; too high overshoots peaks, too low gets stuck.
- Elitism — the fraction of top performers that survive unchanged into the next generation (exploitation vs exploration).
- Evolutionary vs Random — evolutionary NAS breeds from the best-found designs each generation; random search re-samples the whole population from scratch, ignoring past results — watch it converge far slower for the same evaluation budget.
This is the core idea behind Neural Architecture Search (NAS):
instead of a human hand-designing a network, a search algorithm
proposes architectures, trains/evaluates a proxy of each one,
and uses those results to bias where it looks next — evolutionary
algorithms, reinforcement learning and Bayesian optimization
(Optuna's default) are the three dominant strategies in production
AutoML systems today.