A pre-trained network's weights θ split into a frozen set (kept exactly as pre-trained) and a trainable set (updated on the new/target domain). Which weights fall into each set depends on the strategy:
Frozen weight: θ stays = θ_pretrained (no gradient applied)
Trainable weight: θ ← θ − η · ∇θ L_target(θ) (standard SGD step)
LoRA adapter (backbone stays fully frozen):
W' = W₀ + (α/r) · B·A only A, B trained, r ≪ d
⇒ a tiny fraction of parameters move
- Full fine-tune — unfreezes everything above "Freeze depth"; most flexible, most trainable parameters, highest overfitting/forgetting risk with little data.
- Feature extraction — freezes the whole backbone, trains only the final head; fast and cheap but capped in how far it can close a large domain gap.
- LoRA — backbone frozen, small low-rank adapters (A·B) inserted per layer; near full-fine-tune quality from a tiny trainable-parameter budget.
- Domain shift — how different the target domain is from the source the model was pre-trained on; larger shift ⇒ higher starting loss and a harder floor to reach.