📊 Information Theory in Hyperparameter Optimization

About this simulation

Bayesian hyperparameter optimization treats the (unknown) mapping from hyperparameters — here, learning rate and regularization strength — to validation accuracy as a random function. A surrogate model (a Gaussian-process-style estimator) maintains a predictive distribution over that function everywhere in the search space. Information theory gives a principled way to decide where to evaluate next: pick the point whose outcome would most reduce our uncertainty about where the true optimum lies.

How it works

• Predictive entropy H(x) — for a Gaussian predictive distribution, H(x) = ½·ln(2πe·σ²(x)). It grows with the surrogate's variance σ²(x), which shrinks near already-evaluated configurations and grows in unexplored regions.
• Expected information gain I(x; f*) — an entropy-search-style acquisition score that weights H(x) by how likely that region is to contain a better optimum, approximating the mutual information between an observation at x and the location/value of the global optimum f*.
• Every click evaluates the hidden validation-accuracy surface at that (learning rate, regularization) pair, adds observation noise σ, and updates the kernel-smoothed surrogate mean and variance everywhere.
• The surface height always encodes entropy; the color encodes whichever field is selected, so you can see exploration (tall, cool) vs. promising exploitation (colored bright near good trials) at a glance.

Controls

• Kernel lengthscale — how far a single trial's information "reaches" across the space. Short lengthscales make the surrogate trust only nearby points; long ones generalize further from fewer trials.
• Observation noise σ — how noisy each validation-score measurement is, which sets the floor on how low predictive entropy can ever get.
• Surface field — toggle the color mapping between raw predictive entropy and the information-gain acquisition score used to choose the next trial.
• 🎲 Suggest next trial — auto-evaluates the point with the current highest acquisition score, mimicking what an entropy-search optimizer would pick.
• ↺ Reset search — clears all trials and starts over. Drag to rotate, scroll to zoom.

Did you know?

Methods like Entropy Search, Predictive Entropy Search and Max-value Entropy Search formalize exactly this idea: instead of just chasing the best-seen value (exploitation) or the most uncertain point (pure exploration), they query wherever an observation is expected to shrink the entropy of the optimizer's belief about the optimum the most — often finding good hyperparameters in far fewer trials than grid or random search.