This is a live bilevel optimization problem — the kind that underlies
gradient-based hyperparameter tuning. An inner problem fits weights
w to a training loss that depends on a hyperparameter λ
(here, an L2 regularization strength). An outer problem then judges
those weights on a separate validation loss and adjusts λ itself using
gradient descent — but to do that, it needs the hypergradient
dL_val/dλ, the sensitivity of the validation loss to a hyperparameter it
never touches directly.
L_train(w, λ), an anisotropic quadratic plus an L2 penalty 0.5·λ·‖w‖².w and settles at the closed-form minimizer w*(λ).∇_w L_train(w*,λ)=0 gives dw*/dλ without unrolling the whole inner optimization.w*(λ).λ ← λ − η·dL_val/dλ.Unrolled differentiation through hundreds of SGD steps is memory-hungry, which is why practical hypergradient methods (implicit differentiation, conjugate-gradient solvers, or approximate Neumann series) avoid storing the whole inner trajectory — exactly the shortcut the implicit function theorem gives us here.
A live bilevel optimization scene: an inner training loss landscape reshaped by a regularization hyperparameter λ, a gradient-descent ball settling at the analytic minimizer w*(λ), and a hypergradient computed via the implicit function theorem to update λ itself.
The purple bowl is the training loss for weights w given λ. Because w*(λ) has a closed form, differentiating its stationarity condition yields dw*/dλ exactly — the core trick behind implicit-differentiation hypergradients, avoiding unrolling the whole inner optimization.
Drag λ to reshape the bowl and watch the ball re-descend. Take a hypergradient step to run one real outer-loop update of λ using dL_val/dλ, exactly as gradient-based hyperparameter optimization does in practice.
Full unrolled backpropagation through an inner optimizer needs memory proportional to the number of inner steps; implicit differentiation instead solves one linear system at the optimum, which is why it scales to hyperparameter tuning of deep networks.