Discriminative Fine-Tuning: Layer-Wise Learning Rates

Fine-Tuning Setup

Base learning rate η_max 2.0e-3

Discriminative ratio (per-layer decay) 2.6×

Frozen backbone layers 2

LR Schedule

Training State

Step / schedule progress

0 / 260 (epoch 1)

Task loss

1.00

Source retention

100%

Head layer LR (top)

0.0e0

Deepest active LR

0.0e0

How it works

Fine-tuning a pretrained network works best when different layers move at different speeds: shallow backbone layers already hold general features and should barely change, while the new task-specific head needs large updates. Discriminative fine-tuning (Howard & Ruder, ULMFiT 2018) assigns each layer l its own rate:

η_l = η_max · d^(-(L-1-l))
  L = number of backbone layers, l = 0 (deepest) .. L-1 (shallowest)
  d = discriminative ratio (paper default 2.6)

On top of that, the whole schedule is modulated in time by a slanted triangular learning rate (STLR): it ramps up fast for a short warm-up fraction cut_frac of training, then decays slowly for the rest, so the model explores before it settles:

cut = T · cut_frac                    (cut_frac = 0.1 here)
p(t) = t/cut                if t < cut
     = 1 − (t−cut)/(cut·(1/cut_frac−1))   otherwise
lr_mult(t) = (1 + p·(ratio−1)) / ratio     (ratio = 32)
η_l(t) = η_max · d^(-(L-1-l)) · lr_mult(t)

Base LR / discriminative ratio — set η_max and how much slower each deeper layer trains; a ratio near 1 makes every layer update equally (more forgetting risk), a large ratio protects the backbone.
Frozen backbone layers — the bottom-most layers are locked to η = 0, exactly like freezing early convolutional/attention blocks during fine-tuning.
Schedule toggle — compare the STLR curve against a flat constant rate at η_max.
The glow pulsing through each slab is proportional to its live effective LR; the new task head (magenta, top) always trains at full rate since it starts from random weights, and source retention falls when too much learning signal reaches the general-purpose lower layers — the classic stability/plasticity trade-off behind transfer learning.

New task head Backbone (bright = high LR) Frozen layer

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.