Transfer and Fine-Tuning¶

What This Is¶

This page is about one adaptation decision:

given a pretrained backbone, how much of it do I update, and with what budget

"Transfer" is the broad act of reusing a pretrained model. "Fine-tuning" is the narrower act of continuing training on a new task. The working skill is choosing between four adaptation depths — feature extraction, partial unfreezing, full fine-tune, and parameter-efficient adaptation — for the smallest defensible change.

The bad habit this page is trying to prevent is reaching for a full fine-tune before a frozen-feature baseline has been measured.

When You Use It¶

a pretrained model exists for your modality or close to it
labeled data for your task is modest (dozens to low thousands)
you need to move fast and cannot train from scratch
compute is limited and you need to choose the smallest update that holds
the pretrained model's behavior is mostly right but shifts on your slice

The Four Adaptation Depths¶

Ranked from cheapest to most invasive:

Frozen feature extraction — forward pass through the backbone, train only a new head
Partial unfreezing — freeze early layers, train later layers + head
Full fine-tune — update every parameter, usually with a very low learning rate
Parameter-efficient fine-tuning (PEFT) — freeze the backbone, inject small trainable adapters (LoRA, adapters, prefix tuning, IA³)

Decision ladder:

start with frozen feature extraction — the dummy baseline of adaptation
if it underfits, try partial unfreezing of the top N blocks
if that still underfits and the dataset is medium-sized, try full fine-tune with a small LR
if compute or memory is the bottleneck, or you need many task-specific variants, try PEFT

Every step up the ladder adds training cost, overfitting risk, and deployment complexity. Defend each step with a validation-score delta, not with a feeling.

Frozen Feature Extraction¶

import torch.nn as nn
from torchvision.models import resnet50, ResNet50_Weights

backbone = resnet50(weights=ResNet50_Weights.DEFAULT)
for p in backbone.parameters():
    p.requires_grad = False

backbone.fc = nn.Linear(backbone.fc.in_features, num_classes)
# only backbone.fc is trainable

What to inspect:

how many parameters are trainable (sum(p.numel() for p in model.parameters() if p.requires_grad))
validation accuracy relative to a simple dummy baseline
whether the head alone is memorizing — if so, the backbone's features are not matched to your task

Frozen feature extraction works best when the pretraining data looks similar to yours (ImageNet → animal photos) and struggles when domains diverge (ImageNet → medical grayscale).

Partial Unfreezing¶

Unfreeze the later blocks — they encode the most task-specific features and benefit most from adaptation:

for name, p in backbone.named_parameters():
    p.requires_grad = name.startswith("layer4") or name.startswith("fc")

The rule of thumb: in a CNN, unfreeze the last stage first; in a transformer, unfreeze the last few blocks first; the embedding layer should usually stay frozen unless the tokenizer also changed.

Layer-Wise Learning Rates¶

When you do update multiple layers, give the earlier layers a smaller learning rate. They carry more general features and should drift less:

def param_groups(model, base_lr=1e-4, decay=0.75):
    # named_parameters() returns input-side layers first, head last.
    # Iterating in reverse walks head -> input. The head gets base_lr;
    # each earlier (input-side) parameter gets multiplied by `decay` (<1)
    # so embeddings drift least and the head drifts most.
    groups, lr = [], base_lr
    for name, param in reversed(list(model.named_parameters())):
        if not param.requires_grad:
            continue
        groups.append({"params": [param], "lr": lr})
        lr *= decay
    return groups

Sanity check: with base_lr=1e-4 and decay=0.75, the head receives 1e-4, the next layer back receives 7.5e-5, the layer after that 5.6e-5, and so on toward the embedding. If you print the first and last groups' lr and the head value is not the largest, the iteration order is reversed — fix that first.

Discriminative learning rates (or "LR decay by depth") are a well-known technique (ULMFiT, ELECTRA). They make full fine-tunes more robust on small datasets.

Typical choices:

decay factor: 0.65–0.9
base LR at the head: 1e-3 to 1e-4
base LR at the deepest backbone layer: 10–100× smaller than the head

Freezing Schedules¶

Instead of freezing static layers, unfreeze gradually:

All-at-once: simplest. Freeze N top layers, train until validation stalls, then unfreeze more. Repeat.
ULMFiT-style gradual unfreezing: unfreeze one layer at a time, train one epoch per unfreeze.
Warmup + full: freeze backbone for the first K epochs while the head stabilizes, then unfreeze everything and continue with a lower LR.

Warmup + full is the default for most practical vision and NLP fine-tunes. It prevents the randomly-initialized head from flooding the backbone with noisy gradients.

Full Fine-Tune¶

for p in backbone.parameters():
    p.requires_grad = True

optimizer = torch.optim.AdamW(
    backbone.parameters(),
    lr=1e-5,                 # much smaller than from-scratch training
    weight_decay=0.01,
)

Rules:

LR is typically 10–100× smaller than what you would use training from scratch
short schedules — 3 to 10 epochs on most datasets
use early stopping on validation; full fine-tunes overfit fast
pair with LR warmup (500–1000 steps) to protect the pretrained weights

Parameter-Efficient Fine-Tuning (PEFT)¶

PEFT adds a small trainable surface on top of a frozen backbone. The best-known is LoRA.

LoRA¶

Insert a low-rank update ΔW = A B alongside each large matrix W:

import torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r=8, alpha=16, dropout=0.05):
        super().__init__()
        self.base = base
        for p in self.base.parameters():
            p.requires_grad = False
        self.A = nn.Parameter(torch.zeros(r, base.in_features))
        self.B = nn.Parameter(torch.zeros(base.out_features, r))
        nn.init.kaiming_uniform_(self.A, a=5 ** 0.5)
        # B stays zero → initial LoRA contribution is zero, safe warm start
        self.scale = alpha / r
        self.drop = nn.Dropout(dropout)

    def forward(self, x):
        return self.base(x) + self.drop(x) @ self.A.T @ self.B.T * self.scale

The trainable parameters are A and B, which together have r * (in + out) entries — usually <1% of the original matrix.

Which layers to adapt¶

For transformers, the well-established choices are:

attention Q/V projections — strong default (and minimum-viable LoRA)
attention K and output projection — extra coverage, marginal gains
feed-forward layers — necessary for more disruptive style shifts

Starting point: apply LoRA to Q and V only. Add more targets only when the validation score plateaus.

Adapter tuning¶

Insert a small bottleneck MLP inside each block:

x → down_proj (d → d/k) → nonlinearity → up_proj (d/k → d) → + residual

Less popular than LoRA in recent work because LoRA matches quality with fewer parameters and no architecture change.

Prompt tuning / prefix tuning¶

Instead of modifying the weights, prepend trainable "virtual tokens" to the input. Extremely parameter-efficient, competitive at scale, sometimes less stable on small datasets.

Merging adapters¶

After training, LoRA's A B update can be folded into the base matrix: W_new = W + scale * A^T B^T. This removes the inference-time overhead. For multi-task setups you keep them separate and load them per task.

Rank And Scale Choices For LoRA¶

r (rank): 4–16 for most tasks, 32–64 for strong domain shifts
alpha: typically 2× r; interacts with the learning rate
target layers: Q and V first, then K/output, then FFN
learning rate: higher than a full fine-tune (1e-4 to 1e-3), because only a tiny surface is updated

A useful ablation: train the same task at r ∈ {4, 8, 16, 32} and plot validation score vs. trainable parameter count. The sweet spot is usually obvious.

Cross-Domain Transfer¶

The pretrained domain and the target domain are not always aligned. Three common cases:

close domain (ImageNet → animal photos, generic text → news): frozen features work; LoRA or partial unfreeze is plenty
distant domain (ImageNet → medical X-rays, generic text → legal documents): head-only usually underfits; partial unfreeze or full fine-tune often needed
different modality (using a text model for code, or vision for audio spectrograms): results are more variable; start with a baseline trained from scratch before trusting the transfer

Diagnostic: compare frozen-feature performance against a small from-scratch model. If the frozen pretrained model barely beats a tiny from-scratch one, the domain gap is large.

Data Scale vs Adaptation Depth¶

Rough heuristic table:

Dataset size	Sensible first choice
<100 labeled examples	frozen features + head, or few-shot prompting (LLMs)
100–1,000 examples	frozen features + head, or LoRA
1,000–10,000 examples	partial unfreeze or LoRA with more targets
10,000–100,000 examples	full fine-tune with discriminative LR
>100,000 examples	full fine-tune, possibly with continued pretraining

These are defaults, not laws. Always compare adjacent rows before committing.

Training Stability¶

Fine-tuning is less forgiving than training from scratch because the weights you are nudging are already useful. Protective measures:

LR warmup for 500–1000 steps before the main schedule
Gradient clipping at a modest norm (1.0 is typical)
Low LR on the backbone, higher on the head
Regularization: a small weight decay (0.01), dropout only where it was present in pretraining
Early stopping on validation — fine-tunes often overfit within 3–5 epochs

If validation loss goes up within the first epoch, your LR is too high or your head is too cold; add warmup or freeze the backbone for one epoch first.

Evaluating An Adaptation¶

Do not compare fine-tunes on training loss. Compare them on:

validation metric under a fixed split (same one used for the baseline)
weak-slice performance — a fine-tune that wins overall but drops on the minority slice has narrowed behavior
calibration — did the probabilities stay reasonable, or did the model become overconfident?
out-of-distribution robustness — how does it behave on held-out shifts?

A fine-tune that wins only on the average metric is a qualified win.

What To Inspect¶

how many parameters are trainable
the ratio of head LR to backbone LR
the freezing schedule, explicitly
validation vs training loss gap — widening means overfitting starting
LoRA rank sweep results if PEFT is used
weak-slice score alongside the headline metric
calibration before and after adaptation

Failure Pattern¶

Reaching for a full fine-tune before a frozen-feature baseline has been measured. The frozen baseline is your floor. If it is already 90% of the way there, the full fine-tune is paying for a small gain with a lot of risk.

Another common failure: fine-tuning with the same learning rate used for training from scratch. Pretrained weights drift too fast and the model "forgets" what it learned.

A third: fine-tuning on too little data and calling it specialization. Under 500 examples, a fine-tune is often just noise around the pretrained baseline.

Common Mistakes¶

skipping the frozen baseline
using a from-scratch learning rate for a fine-tune
changing the tokenizer or data format without resetting input embeddings
updating backbone and head with the same LR on small data
not using LR warmup on full fine-tunes
evaluating on training-data-like validation that does not capture the real shift
applying LoRA to only one layer type and concluding LoRA does not work for your task
merging LoRA adapters before finishing evaluation (lose the ability to ablate cleanly)
treating fine-tuning as a substitute for better data
assuming a model that beat the baseline on average also improved every slice

Practice¶

Train a frozen feature extractor + head on a small dataset and report its validation accuracy. This is your floor.
Unfreeze the last block and retrain. Report the delta and whether it was worth the compute.
Run a full fine-tune with and without LR warmup. Report which one was more stable in the first epoch.
Add LoRA adapters to Q and V of a pretrained transformer. Compare against full fine-tune on validation and on weak slices.
Sweep LoRA rank over {4, 8, 16, 32} and plot trainable parameter count vs validation score.
Fine-tune on a slice of the data (100, 500, 2000 examples) and show the scale vs adaptation-depth trade-off.
Evaluate a fine-tuned model on a domain-shifted slice and report whether the adaptation helped or hurt it.
Merge a trained LoRA into the base weights and verify that inference output matches the unmerged version.

Runnable Example¶

This example stays local-only for now because the browser runner does not yet include PyTorch.

Longer Connection¶

Continue with Optimizers and Regularization for the optimizer-level choices that fine-tuning depends on, Attention and Transformers for the target architecture most fine-tunes are applied to, and Optimization, Regularization, and PEFT for the full adaptation workflow as a track.