Clinic 16

PEFT Depth Or Full Fine-Tune

A 7B base model, 8k domain examples, one A100 for a week. LoRA, last-n-layers, or full fine-tune — defend one under the compute cap. The easy answer is not the right one.

Back To Clinics Open PEFT Topic Open Transfer & Fine-Tuning

Situation

A Fixed Compute Budget

7B base, 8k curated domain pairs, one A100 for seven days. Deploy something useful by Friday.

Your Job

Pick A Fine-Tuning Depth

LoRA at rank 8, last-4-layer unfreeze, or full fine-tune. Three different compute profiles, three different failure modes.

Bad Habit To Avoid

When Depth Trades Against Reps

A deeper adaptation gets one shot at the budget. A shallower one gets several. Decide which gives you the better posterior over runs, not the better single-run upside.

Situation

Your team has an open-weights 7B-parameter decoder language model that works reasonably on the domain. You have:

• base model: 7B dense decoder, bf16 weights (~14 GB on disk)
• training data: 8000 curated (prompt, response) pairs from a specialty domain (legal contract review). Each ~400 tokens average.
• compute: one A100 80GB for 7 days. Nothing else.
• goal: deploy a fine-tuned model that meaningfully improves on the base model for contract-review queries

Three options:

• LoRA at rank 8, targeting q_proj, v_proj in every transformer block
• last-4-layer unfreeze — freeze the bottom 28 layers, full-fine-tune the top 4
• full fine-tune — everything learns

Your evaluation set is 200 held-out contract-review questions with reference answers and a legal reviewer who scores correctness on a 5-point scale.

Artifact Packet

option	trainable params	VRAM (fwd+bwd)	typical 7B epoch wall-clock	checkpoint size	regression risk	reversibility
LoRA r=8	~8M (0.1%)	~24 GB with qlora, ~50 GB bf16	~4 h / epoch	~30 MB adapter	low — base weights frozen	high — just unload the adapter
Last-4-layer FT	~0.5B (~7%)	~70 GB bf16, needs careful batch	~9 h / epoch	2 GB (partial) or 14 GB (full copy)	medium — top layers shift	medium — need the old layers around
Full fine-tune	7B (100%)	≥ 80 GB — borderline for A100	~18 h / epoch + optimizer state	14 GB	high — can regress on out-of-domain	low — you own the new weights

Additional facts to read carefully:

• the team has no existing RAG setup for legal documents; the fine-tune is supposed to absorb the pattern, not the citations
• the evaluation set is fixed; a held-out 100-question out-of-domain set (general coding questions) is used as a regression guard
• the legal reviewer is available for 2 hours of grading this week, total — so the number of candidate models you can score end-to-end is bounded
• the 8k pairs are curated, not scraped — high-quality but small

Decision Prompt

Write a six-sentence defense that answers:

1. Which option do you pick, and why under this compute cap?
2. How many epochs are feasible in the 7-day window for your chosen option?
3. What is your single inspection during training — a loss curve, a periodic generation, a held-out reviewer sample?
4. What regression guard catches out-of-domain damage early?
5. What would make you switch to a different option next cycle?
6. If the LoRA is chosen and underperforms, is the next move "higher rank" or "last-n unfreeze"?

Strong Reasoning Looks Like

• picks LoRA r=8 (or r=16 with QLoRA) for this budget — it is the only option that lets you run multiple full training passes in 7 days, iterate on hyperparameters, and still have reviewer time left for evaluation
• calls the compute: 7 days × ~4 h/epoch = ~40 epochs achievable for LoRA; 7 days × ~9 h/epoch = ~18 epochs for last-4-layer; full fine-tune eats the entire budget on a single 7-epoch run with no room to iterate
• names the inspection: training loss plus a held-out 20-question mini-eval (auto-scored against reference answers with ROUGE or BLEU) every 200 steps; the legal reviewer only sees the final two candidate models
• defends the regression guard: run the 100-question OOD set on the base model, the best LoRA checkpoint, and optionally the last-4-layer model; if LoRA drops OOD score by more than 2 points, stop
• treats last-4-layer as the escalation path — if LoRA at rank 8 / 16 / 32 all plateau below the reviewer bar, then last-4-layer is the next controlled step
• explicitly de-prioritizes full fine-tune — it wastes the compute budget on one run and carries the highest regression risk

Common Wrong Moves

• picking full fine-tune because "we have the data and we own the compute" — one run fills the budget; there is no room to iterate or reverse
• picking LoRA at rank 256 thinking bigger is better — at high ranks LoRA loses most of its compute advantage without clearly matching a partial FT
• picking LoRA but only adapting q_proj without v_proj — the academy's PEFT and LoRA topic is explicit that q+v is the sane starting pair
• running full fine-tune without saving intermediate checkpoints and discovering at day 6 the loss went to NaN at hour 40
• evaluating by only the contract-review score and missing a silent OOD regression
• letting the reviewer's limited time make you pick "the one I can explain", not "the one that wins under the cap"

Run The Clinic In Browser

Use the runner to sketch the compute budget spreadsheet and check the arithmetic on epochs-per-option.

Reference Reveal

Open only after you write the defense

The reference choice is **LoRA r=8 targeting q_proj and v_proj, QLoRA if VRAM is tight, 3-5 epochs per run, and 3 candidate runs during the week**. Why: - **compute cap is the forcing function**. Full fine-tune gets one shot. LoRA gets several, plus headroom to ablate rank, target modules, and learning rate. On a fixed eval budget (200 questions + 2 reviewer-hours), multiple candidates is a bigger win than one "best" candidate. - **8k pairs is a small dataset** for full fine-tuning. The model will overfit the specific phrasing of the curated pairs and generalize less well than a LoRA that only perturbs a tiny fraction of parameters. - **deployment reversibility matters**. A LoRA adapter is a 30 MB file you can swap in and out; a fine-tuned 14 GB checkpoint is a commitment. For legal-domain deployments, being able to point at exactly what changed is an audit feature. - **q+v target modules** is the academy's default starting pair; the PEFT topic explains the rationale (k+o and MLP are the escalation path, not the starting point). Proposed schedule: - days 1–2: three LoRA runs at different learning rates (1e-4, 3e-4, 1e-3), 3 epochs each. Mini-eval between them. - days 3–4: a rank-16 run with the best learning rate; also a rank-8 run with q+v+k+o. Mini-eval each. - day 5: pick the best two candidates; run full eval with reviewer on both. - days 6–7: last-layer unfreeze as an escalation if none of the LoRA candidates clear the reviewer bar. Escalation rule: - if the best LoRA scores < reviewer's 3.5/5 bar on contract-review AND regresses on OOD by > 2 points, you have time to try last-4-layer unfreeze in days 5–6. - if the best LoRA clears 3.5/5 on contract-review without OOD regression, ship it. Save the remaining compute for a second domain or a rank ablation. Switch criteria (for next cycle): - LoRA plateaus below the reviewer bar across three rank-and-target-module configurations → escalate to last-4-layer - last-4-layer still plateaus → the data quantity (8k pairs) is the real bottleneck; fix that before paying for full fine-tune - training loss goes to NaN → smaller learning rate, not smaller rank The practical lesson: **the compute cap picks your fine-tuning depth**. Most organizations that "need full fine-tuning" actually need either (a) more data, (b) better retrieval, or (c) a higher-rank LoRA they did not try.

What To Do Next

1. open PEFT and LoRA for the math and the target-module priority
2. open Transfer and Fine-Tuning for freezing schedules and layer-wise LR
3. open Freeze Or Fine-Tune? for the adjacent classical clinic
4. draft the compute spreadsheet for your own hardware and data size; if the math disagrees with this clinic's reference, the clinic's answer flips