Variational Inference: Watching a Posterior Get Approximated

Target Posterior

Variational Inference

Learning rate 0.15

Samples per step (K) 12

Live Fit

ELBO

—

Step

Mean σ

1.00

How it works

Variational inference replaces exact Bayesian sampling with optimization: pick a simple family q_φ(θ) — here a diagonal Gaussian N(m, diag(σ_x², σ_z²)) — and push its parameters φ = (m, log σ) to make q as close as possible to the true (unnormalized) posterior p̃(θ) shown as the 3D landscape.

ELBO(φ) = E_q[log p̃(θ)] − E_q[log q(θ)]
maximizing ELBO  ⇔  minimizing KL(q ‖ p)

The entropy term has a closed form for a diagonal Gaussian: H[q] = log σ_x + log σ_z + log(2πe). The first term is estimated with K Monte-Carlo samples drawn via the reparameterization trick, θ = m + σ⊙ε with ε ~ N(0,1), so gradients can flow straight through the sample:

∇_m ELBO ≈ (1/K) Σ ∇_θ log p̃(θ_k)
∇_logσ ELBO ≈ (1/K) Σ ∇_θ log p̃(θ_k)·ε_k·σ  +  1

Each optimization step draws K fresh samples, computes these two gradients analytically from the chosen target's exact log-density, and takes a gradient-ascent step on (m, log σ). The green ellipse is the current 2σ contour of q; the small dots are live samples drawn from q, settling onto the landscape as it converges.

Target select — switches the true log-posterior: a single correlated mode, two separated modes, or a curved "banana" posterior (the classic case where a Gaussian must compromise).
Learning rate — step size of the gradient-ascent update.
Samples per step (K) — more samples give a less noisy gradient estimate but cost more compute per step.
Restart — reinitializes q at a random location with σ = 1 and resets the step counter.

Real-world relevance: this is the exact mechanism behind mean-field / ADVI-style variational inference used in probabilistic programming (Stan's ADVI, Pyro, PyMC) to approximate posteriors too expensive to sample with full MCMC. It also demonstrates VI's known failure mode: on the bimodal target, q collapses onto a single mode instead of covering both (KL(q‖p) is mode-seeking, unlike MCMC which explores every mode it can reach).

True posterior p̃(θ) (log-density surface) Variational mean m & 2σ contour Live samples from q(θ)

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.