Discriminative Fine-Tuning: Layer-Wise Learning Rates (2D)

Fine-Tuning Setup

Base learning rate η_max 2.0e-1

Discriminative ratio (per-layer decay) 2.6×

Frozen backbone layers 1

LR Schedule

Training State

Step / schedule progress

0 / 240 (epoch 1)

Discriminative loss

0.693

Uniform loss

0.693

Loss gap (uniform − discriminative)

+0.000

Source retention

100%

Head layer LR

0.0e0

How it works

A real 4-layer network (2 → 8 → 8 → 8 → 1, tanh hidden units, sigmoid head) is being fine-tuned on a genuine two-moons binary classification task via true backpropagation and gradient descent — every step here is a real forward pass, a real loss, and real weight updates, run twice from the same starting weights so the two strategies are compared fairly:

η_l = η_max · d^(-(L-1-l))
L = 4 weight layers, l = 0 (earliest/deepest) .. L-1 (head)
d = discriminative ratio

The whole schedule is modulated by a slanted triangular learning rate (STLR): fast warm-up for a short fraction of the run, then a slow decay:

cut = T · 0.1
p(t) = t/cut                          if t < cut
     = 1 − (t−cut)/(cut·9)            otherwise
mult(t) = (1 + p·31) / 32
η_l(t) = η_max · d^(-(L-1-l)) · mult(t)

Discriminative net applies η_l(t) to each of its 4 weight layers separately (deepest layers can also be fully frozen at η = 0).
Uniform net is cloned from the exact same initial weights and trained on the exact same data, but every layer shares one flat rate η_max·mult(t) — the classic "one learning rate for the whole network" fine-tuning approach.
Each panel's shaded background is the network's actual decision boundary, recomputed every frame from a real forward pass over a grid — not a canned image.
Source retention measures how far the earliest weight layer has actually drifted (Frobenius norm) from its pretrained starting point — the real stability/plasticity trade-off behind transfer learning.

Discriminative net Uniform-LR net Class boundary / points