PEFT and LoRA¶

What This Is¶

Parameter-Efficient Fine-Tuning (PEFT) is the family of methods that adapt a pretrained model without updating all of its weights. LoRA (Low-Rank Adaptation) is the most important member of that family today. The premise is simple: a good adapter is a small perturbation of a large base model.

This topic is a deeper treatment of the PEFT surface introduced in Transfer and Fine-Tuning. Where that topic covers the four-depth adaptation ladder end to end, this topic goes one level down into the PEFT design choices: which layers to adapt, what rank to pick, how to compose multiple adapters, and how to inspect one.

When You Use It¶

you have a pretrained model worth gigabytes and an adaptation task worth megabytes of new parameters
you need to serve many fine-tunes of the same base model without paying GPU memory for each
you want to switch domain/persona/task by swapping an adapter file rather than loading a new model
you have a small-to-medium dataset (10³–10⁵ examples) and full fine-tuning will overfit
compute is the binding constraint and a full fine-tune would not finish inside your budget

Do Not Use It When¶

the task differs structurally from pretraining (e.g., adapting a text LLM to predict tabular regressions) — PEFT cannot rescue the wrong inductive bias
training data is on the order of pretraining data — full fine-tuning will generalize better
the final deployment target is weight-merged and latency-critical and your library does not support merge (most do)

PEFT Taxonomy¶

method	what it inserts	typical params	notes
LoRA	low-rank update `ΔW = BA` on chosen linear layers	0.1–3% of base	the production default
QLoRA	LoRA on a 4-bit quantized base	same as LoRA	lets you fine-tune larger models on consumer GPUs
DoRA	decomposes weight into magnitude + direction, LoRAs the direction	slightly > LoRA	a recent strong drop-in
Adapters (Houlsby, Pfeiffer)	bottleneck MLP inserted between transformer sublayers	1–5% of base	older; LoRA usually beats them per parameter
Prefix / Prompt tuning	learned "prompt" tokens prepended to every layer's keys/values	0.01–0.1% of base	cheap, weaker than LoRA on hard tasks
IA³	learned per-channel gates on activations	0.01% of base	extremely cheap, limited expressiveness
BitFit	train only bias terms	0.01–0.1%	surprisingly strong baseline for small tasks

For a new project: start with LoRA. If base-model memory is the constraint, start with QLoRA. The rest are worth knowing but rarely the default.

LoRA — The Math In One Paragraph¶

For a frozen pretrained weight W ∈ ℝ^{d_out × d_in}, LoRA learns two small matrices A ∈ ℝ^{r × d_in} and B ∈ ℝ^{d_out × r} with rank r << min(d_in, d_out). The adapted weight at inference is:

W' = W + (α / r) · B A

Key facts this encodes:

parameters: only r(d_in + d_out) instead of d_in · d_out
A is initialized with small random values; B is initialized to zero, so the adapter starts as an identity map
α is a scaling hyperparameter — a common default is α = 2r or α = r; changing it is approximately equivalent to changing the effective learning rate on the adapter
at serving time you can merge: W ← W + (α/r) B A, and there is no inference cost; the merge is reversible

LoRA In PyTorch — A Self-Contained Implementation¶

The canonical form, readable in fewer than 40 lines:

import torch
from torch import nn

class LoRALinear(nn.Module):
    """Low-rank update around a frozen nn.Linear."""

    def __init__(self, base_linear: nn.Linear, r: int = 8, alpha: int = 16, dropout: float = 0.0):
        super().__init__()
        self.base = base_linear
        for p in self.base.parameters():
            p.requires_grad = False

        in_f = base_linear.in_features
        out_f = base_linear.out_features

        self.A = nn.Parameter(torch.randn(r, in_f) * (1.0 / r**0.5))  # small random init
        self.B = nn.Parameter(torch.zeros(out_f, r))                  # zero so ΔW = 0 at step 0
        self.scale = alpha / r
        self.dropout = nn.Dropout(dropout) if dropout > 0 else nn.Identity()

    def forward(self, x):
        return self.base(x) + self.dropout(x @ self.A.T) @ self.B.T * self.scale

The init is a symmetry: identity-at-step-0 only requires that BA = 0, so exactly one of A or B must be initialized to zero. Either pattern works — random A, zero B (used here) or zero A, random B (you will see this elsewhere). If both are random the adapter is non-zero at step 0; if both are zero the adapter never starts learning.

Two inspections you should run the first time you use this:

the output at step 0 must equal self.base(x) exactly (because B = 0). If not, you have a bug.
sum(p.numel() for p in self.parameters() if p.requires_grad) must be exactly r(in_f + out_f). If it is larger, your base is accidentally trainable.

Which Layers To Adapt¶

Not every linear layer is equal. On transformer architectures, priority order from strongest per-parameter return:

q_proj, v_proj in attention — the canonical choice from the original LoRA paper; small, strong
add k_proj, o_proj — modest gains, still cheap
add the MLP projections (gate_proj, up_proj, down_proj in modern architectures) — expands capacity further
add token embeddings or LM head — almost never worth it unless you are changing vocabulary

The default heuristic: {q_proj, v_proj} first. Expand to {q, k, v, o} if you have headroom. Only add the MLPs if the evaluation is still moving when you do.

A common mistake is LoRAing every linear layer by default. It blows up the adapter parameter count and frequently reduces accuracy relative to the careful set above.

Rank And Alpha — The Two Knobs¶

rank `r`	scale `α`	behavior
4	8	tiny, strong floor for simple task / style adaptation
8	16	production default — works on most instruction-tuning tasks
16	32	when the task is harder or data is larger; diminishing returns after
32+	64+	at this point full fine-tune is often cheaper than a large LoRA

Tuning advice:

tune r first. Double it until validation stops improving.
hold α / r fixed at 2 — this keeps the effective LR sensible as you grow r.
if validation loss is unstable, reduce α / r rather than reducing the base LR.

Multiple Adapters, Merged And Unmerged¶

Two deployment shapes worth knowing:

Hot-swap: keep the base frozen, swap LoRA weights per request to serve multiple fine-tunes from one GPU. No merge. This is the common case.
Merged: after training, fold the adapter into the base weights and ship a single model. Zero inference overhead. Good when you only serve one specialization per process.

You can compose adapters additively in special cases — W' = W + ΔW_1 + ΔW_2 — but most composition bugs come from treating this as algebraically safe when the two adapters were trained independently. Default to running one adapter at a time.

Training Recipe¶

A working LoRA recipe for instruction-tuning a 7B-class model on 10⁴–10⁵ examples:

optimizer: AdamW, lr=1e-4 to 3e-4 on the adapter (higher than full fine-tune — only the adapter learns)
weight decay: 0 on the LoRA parameters (they are already low-rank and regularized)
schedule: cosine decay with a 3% warmup, no restart
batch size: effective batch of 32–128 via gradient accumulation
epochs: 1–3 is usually enough; more and you start memorizing
gradient checkpointing: on (you saved memory on weights — spend some of it on longer context)
mixed precision: bf16 if the GPU supports it, else fp16 with loss scaling

If you are using QLoRA (base quantized to 4-bit), keep the adapter itself in higher precision (bf16) and use paged optimizers for the host-side state.

What To Inspect¶

parameter count — confirm it matches r(d_in + d_out) × number_of_adapted_layers. A surprise here means your target_modules setting is wrong.
step-0 loss vs. base model — must match. If it does not, B was not zero-initialized or the forward path is broken.
gradient norms on A vs. B — both should be non-trivial after a few steps. If either stays near zero, the scaling is wrong.
merge test — train, then compare outputs before and after merge. They should agree to numerical precision. A drift means the merge math is wrong for your layer type (common on quantized bases).
adapter size on disk — should be a few MB, not GB. If it is GB, you saved the base by accident.

Failure Pattern¶

The most common LoRA failure is silent convergence to the pretrained model. The loss looks flat; validation looks identical to baseline. This almost always traces to one of:

LR too low (the adapter needs 5–10× the LR of a full fine-tune)
wrong target_modules — e.g., a library naming mismatch means no layers are actually being LoRA'd
frozen flags misapplied — everything is trainable, so you are accidentally doing a full fine-tune with a tiny LR, which goes nowhere

The second failure is train loss drops, validation does not. This is the classic too-large-rank-on-too-small-dataset trap. Cut r in half and rerun.

Common Mistakes¶

setting B to random init instead of zero — breaks the identity-at-init guarantee
setting A to zero — training never starts
merging the adapter at inference without verifying outputs match the pre-merge model
applying weight decay to LoRA parameters — slow, unnecessary
using the same LR for the base and the adapter when both are trainable (they shouldn't both be trainable)
choosing target_modules by name-matching strings in the wrong library (HuggingFace vs. Lit vs. nanoGPT all use different names)
re-tuning on top of a merged adapter — can work, but track the merge chain; otherwise reproducibility breaks

Decision: LoRA vs. Full Fine-Tune vs. Prompt-Only¶

option	when it wins	trade
prompt-only / in-context	task spec fits in context, low volume, or rapidly changing	no training cost; limited steerability
LoRA (PEFT)	10³–10⁵ examples; task style or domain shift; need to serve many variants	small artifacts; must learn the tooling
full fine-tune	10⁵+ examples; deep task shift; single specialization	biggest lift; highest cost; heaviest artifact

A useful rule: if you cannot tell whether LoRA or full fine-tune wins, LoRA is the right call — it is cheaper to iterate, easier to revert, and the evidence will accumulate.

Practice¶

Take a 1B–2B parameter base LLM. Apply the LoRALinear wrapper to q_proj and v_proj. Train for one epoch on a small instruction set. Report the parameter count and the adapter file size.
Run your step-0 output identity check — confirm the model with the fresh adapter matches the base model exactly.
Sweep r ∈ {4, 8, 16, 32} with α = 2r. Plot validation loss against r. Pick the value where the curve bends.
Train two LoRAs for two different personas. Swap them at inference. Confirm the outputs follow the adapter currently loaded.
Merge one adapter into the base. Measure inference latency before and after merge on a fixed prompt — the merged model should be measurably faster than the hot-swap path because the adapter matmul is gone.
Deliberately break the recipe: set B to random, or pick target_modules=[]. Confirm that each break matches the failure signature described above.

Runnable Example¶

LoRA training requires a PyTorch + CUDA environment, which the browser runner does not provide. Run the recipe locally with pip install peft transformers accelerate and the Hugging Face PEFT library, or paste the LoRALinear snippet above into a notebook and train against a small model (e.g., gpt2) on CPU to confirm the identity-at-init property.

Longer Connection¶

PEFT lives at the intersection of three topics already in the academy:

Transfer and Fine-Tuning gives the four-depth adaptation ladder; PEFT is the fourth rung
Optimizers and Regularization — the LR / weight-decay / warmup recipe here is a specialized instance of the full-model recipe, with different defaults because only a small subset of parameters is trainable
Attention and Transformers — knowing where the q_proj/v_proj layers are in a transformer is what lets the "which layers to adapt" section make sense

For the decision surface — LoRA vs. full fine-tune vs. RAG vs. prompt — read Text Generation and Language Models and Retrieval-Augmented Generation back to back. PEFT is one option inside that larger decision; it should never be chosen in isolation.