Clinic 15

Prompt Vs Retrieval Vs Fine-Tune

An internal knowledge-base bot misses on 18% of answers. You have one engineering week. Prompt engineering, retrieval, or fine-tuning — and a budget that forbids two of them.

Back To Clinics Open Prompting Topic Open RAG Topic

Situation

18% Failure, One Week

A working LLM-based helpdesk bot, one persistent error class, three candidate fixes, a very tight budget.

Your Job

Pick One Lever

Prompt, retrieval, or fine-tune. Defend the choice. Name the inspection that will prove the fix actually closed the error class.

Bad Habit To Avoid

Reaching For Fine-Tuning

Fine-tuning is the most expensive lever. Reach for it last, after a clear finding from prompt or retrieval.

Situation

An internal engineering organization deployed a ChatGPT-style bot over their documentation three months ago. It passes most queries well enough. A failure analysis from last week turns up:

• total evaluation queries: 500
• model answers correct and useful: 410 (82%)
• model answers wrong or irrelevant: 90 (18%)

The 90 failures cluster:

• 52 are on questions about internal APIs and services that were added after the base model's training cutoff
• 21 are on questions where the correct answer is in the docs, but the bot invented a plausible-looking wrong one
• 11 are on policy questions where the bot has the right facts but the wrong tone (too casual for a compliance context)
• 6 are malformed requests the bot should have refused or asked to clarify

You have one engineering week and a compute budget that allows exactly one of:

• tighter prompt engineering (prompt templates, few-shot, guardrails)
• retrieval augmentation over the internal docs (chunking, vector store, retrieval-at-answer-time)
• supervised fine-tuning on 2000 cleaned query/answer pairs

Artifact Packet

lever	direct addresses	cost	risk	reversibility
prompt	tone, refusal behavior, format discipline	low (hours)	no effect on knowledge gaps	instant — just change the prompt
retrieval (RAG)	out-of-training internal knowledge, invented-answer on documented facts	medium (days)	adds latency, retrieval can miss	high — disable retrieval to fall back
fine-tune	systematic tone shifts, format adherence, rare patterns	high (compute + data curation)	overfits the 2000 pairs; can regress elsewhere; slow to iterate	low — you own the new weights

Failure-cluster breakdown:

cluster	size	what was wrong with the answer
internal APIs post-cutoff	52	the model did not know about an API documented after its training cutoff
invented answer on documented facts	21	the answer is wrong, but the correct version is in the internal docs
wrong tone on policy	11	the answer is correct but does not follow the team's tone/policy guide
malformed → should have refused	6	the model attempted an answer where a refusal was the right move

Before reading on, classify each cluster yourself: which lever does each cluster's cause live in? Then count which lever moves the most failures and at what cost. The right answer is the one whose cost matches the cluster sizes.

Decision Prompt

Write a six-sentence defense that answers:

1. Which lever do you pick, and for which failure cluster primarily?
2. How does the error budget decompose — if you fix only your chosen cluster, what is the new overall failure rate?
3. What evaluation suite will confirm the fix (regression + target set + out-of-distribution)?
4. What is the single most likely way this lever produces a different failure you have not yet considered?
5. What would make you switch levers (for next week's cycle)?
6. Would you ship this fix on its own or pair it with one cheap prompt tweak?

Strong Reasoning Looks Like

• it picks retrieval and targets the 73 (= 52 + 21) failures driven by missing / unconsulted knowledge — the largest reachable cluster
• it predicts a new failure rate: roughly 17 / 500 = 3.4% remaining if the 73 fail cluster closes, modulo new retrieval-specific failures
• it commits to a two-loop eval: retrieval-recall@k on a curated question set, plus end-to-end answer quality on the original 500
• it anticipates the RAG failure modes: retrieval misses (a legitimate doc was not retrieved), retrieval hits but the model still ignored the context, and new hallucinations that now cite a real doc incorrectly
• it bundles a cheap prompt tweak alongside — a system-prompt policy for refusals addresses the 6-query malformed cluster at no extra compute
• it schedules fine-tuning as "later, if the 11-query tone cluster turns out to resist prompting"

Common Wrong Moves

• picking fine-tuning first because "we need the model to learn our docs" — models do not learn reliably from 2000 pairs; RAG puts the docs in the context without any training
• picking prompt-only because it's cheap — it cannot fix the 73-query knowledge cluster
• building RAG and not measuring retrieval-recall separately from answer-correctness — you will not know which half is broken when the new failures appear
• evaluating only on the 90 failures from last week — the old 410 correct answers might quietly regress under the new RAG pipeline
• ignoring latency: a retrieval-augmented bot that feels slow to users is a deployment regression, not just an eval-metric one
• running all three levers in parallel "just in case" — you lose attribution on the source of any win or loss

Run The Clinic In Browser

Use the runner to sketch a scoring matrix — per-cluster coverage vs. lever — and write the two-loop eval pseudocode.

Reference Reveal

Open only after you write the defense

The reference choice is **retrieval augmentation, paired with one cheap system-prompt tweak**. Why retrieval: - 73 of 90 failures are knowledge-access failures — the docs exist, the model did not see them at answer time. That is the single-cause problem retrieval is designed for. - the fix is directly diagnosable: if retrieval-recall@5 is high but answer-correctness does not move, the retrieval pipeline is working and the model is the issue (not hugging the context). If retrieval-recall@5 is low, the chunking or embedder is the issue. - it is reversible within seconds: disable retrieval and you have the old bot. Why bundle the prompt tweak: - the six malformed-request failures are in scope for a new system-prompt refusal rule. Cost: minutes. - strong system-prompt guidance on "use the retrieved context, quote it, refuse if it is not there" is part of any competent RAG deployment anyway. Why not fine-tune first: - fine-tuning on 2000 pairs is a lot of compute for an uncertain win on the 11-query tone cluster. Tone is the thing prompts *are* good at. - a fine-tuned model that regresses on the 410 correct answers is a very expensive loss. - the clock says one week. You can run the prompt + retrieval lever, measure, and *then* schedule fine-tuning for next cycle if tone still fails. Evaluation plan (two-loop): - **retrieval loop**: curate 150 questions with labeled ground-truth passages. Measure retrieval-recall@5. Target: ≥ 0.9. - **answer loop**: the original 500-query set plus a 50-question OOD set. Primary: answer correctness on the 90 failures. Secondary: no regression on the 410. Tertiary: answer latency distribution. Expected outcome: end-to-end failure rate drops from 18% to 4–6%. The residual failures reveal whether next week's lever should be (a) better retrieval, (b) prompt discipline for context-hugging, or (c) fine-tuning for tone. The practical lesson: **fine-tuning is the most expensive lever and usually not the first one to reach for**. Knowledge-access failures belong to retrieval; tone/format failures belong to prompts; *only* the failures that survive both levers belong to fine-tuning.

What To Do Next

1. open Retrieval-Augmented Generation — the topic with the two-loop evaluation recipe
2. open Prompting and Tool Use — for the system-prompt tweak
3. open PEFT and LoRA — the right fine-tuning tool when fine-tuning is actually the lever
4. draft the error-cluster table on your own bot's eval set; if the clusters are not one-dominated, the clinic's answer flips