Retrieval-Augmented Generation

What This Is

Retrieval-augmented generation (RAG) is the pattern where a generator (usually an LLM) answers a query after being handed retrieved context from a corpus instead of relying only on its training parameters. The retrieval step is a classical information-retrieval problem (embed, index, score); the generation step is conditioned on the retrieved passages.

RAG exists because three things are true at once:

• model parameters are expensive to update and freeze the model's knowledge at training time
• business corpora, documentation, and facts change much faster than model training cycles
• hallucinations are cheaper to catch when the generator must ground its answer in retrieved text

This topic teaches RAG as a workflow decision — when to use it instead of prompt-only or fine-tuning — and as an inspection target with two distinct evaluation loops (retrieval quality vs. grounded-answer quality).

When You Use It

• the ground-truth answer lives in a corpus the model did not see during training (company docs, customer tickets, research PDFs)
• facts change weekly and retraining is not an option
• you need citations so a human reviewer can audit the answer
• the query distribution is narrow enough that a search index earns its keep, but not narrow enough that a fine-tune on the corpus is justified
• you need the freedom to update the corpus without changing the model

Do Not Use It When

• the answer is a stable world fact the model already knows — retrieval adds latency and a new failure surface for no gain
• the corpus is tiny (< 50 documents) and fits in the prompt directly
• the task is reasoning over structured data — SQL or a tool call is a better primitive than text retrieval
• the model's weakness is reasoning, not knowledge — RAG cannot rescue a weak reasoner

The RAG Loop

              ┌──────── indexing (offline) ────────┐
              │                                    │
  corpus ──▶ chunker ──▶ embedder ──▶ vector index │
              │                                    │
              └───────────────┬────────────────────┘
                              │
                              ▼
  user query ──▶ embedder ──▶ retriever ──▶ top-k passages
                                                │
                                                ▼
                              grounded prompt ──▶ generator ──▶ answer + citations

Three decisions determine the quality of the loop:

1. how to chunk the corpus — too big and passages are noisy; too small and they are context-free
2. what to retrieve — cosine on dense embeddings, BM25, or a hybrid rerank
3. what to pass to the generator — top-k, with or without a rerank, with or without compression

Library Notes

• sentence-transformers — fine-tuned dense embedders; all-MiniLM-L6-v2 is the working-class default
• FAISS — the retrieval index for cosine / inner product; fast and well-supported
• rank_bm25 — classical lexical baseline; run it as a sanity check against every dense retriever
• LangChain / LlamaIndex — orchestration frameworks; useful for scaffolding, easy to over-engineer with
• Ragas — evaluation harness specific to RAG (retrieval precision, answer faithfulness, answer relevance)

Prefer a bare FAISS + sentence-transformers stack for the first working version. Add a framework only when you have a real reason.

Minimal Example

A minimal, inspectable RAG pipeline on an in-memory corpus:

from sentence_transformers import SentenceTransformer
import faiss
import numpy as np

# --- Corpus ---
docs = [
    "The academy's Study Plan has six phases.",
    "Decision clinics force a commit-before-reveal habit.",
    "Honest splits must match the deployment story.",
    "IOAI competitions often ship a non-standard grading metric.",
    "Transfer learning reuses features from a pretrained encoder.",
]

# --- Index (offline step) ---
encoder = SentenceTransformer("all-MiniLM-L6-v2")
doc_embeds = encoder.encode(docs, normalize_embeddings=True)
index = faiss.IndexFlatIP(doc_embeds.shape[1])  # inner product == cosine (since normalized)
index.add(np.asarray(doc_embeds, dtype="float32"))

# --- Retrieval ---
def retrieve(query: str, k: int = 2):
    q = encoder.encode([query], normalize_embeddings=True).astype("float32")
    scores, idx = index.search(q, k)
    return [(docs[i], float(s)) for i, s in zip(idx[0], scores[0])]

# --- Grounded prompt ---
def grounded_prompt(query, retrieved):
    context = "\n".join(f"- {d}" for d, _ in retrieved)
    return (
        "Answer the question using only the context. "
        "If the context is insufficient, say so explicitly.\n\n"
        f"Context:\n{context}\n\nQuestion: {query}\nAnswer:"
    )

q = "What do decision clinics teach?"
hits = retrieve(q, k=2)
print(grounded_prompt(q, hits))
# pass to an LLM of your choice

The important move is not the 20-line scaffold. It is that you now have two independent failure surfaces (retrieval and generation) and you can inspect each in isolation.

Chunking Strategy

Chunking determines what the retriever is allowed to find. The three common strategies:

strategy	what it does	when it wins
fixed-window (e.g., 512 tokens, 64-token overlap)	uniform chunks across the corpus	works as a strong default; never catastrophically bad
semantic / recursive	splits on paragraphs, then sentences, then tokens	markdown, code, docs with structure
document-aware (section headers as boundaries)	one chunk per section	technical docs, legal contracts, policy PDFs

Two rules that hold across strategies:

• always include a small overlap between chunks (10–20% of chunk size) so a sentence near a boundary does not get orphaned
• always store the source path + offset with each chunk so the generator can cite it and a human can audit it

Retrieval: Dense, Sparse, Hybrid

Three retrievers and when each is the right call:

• dense (sentence-transformers + cosine) — semantic matching, handles paraphrase well, weak on rare entities (names, numbers, IDs)
• sparse (BM25) — lexical matching, unbeatable on exact-match queries, weak on paraphrase
• hybrid (reciprocal rank fusion of the two) — nearly always the right production answer; costs one extra pass

Run BM25 as a baseline even when you plan to ship dense. If BM25 matches dense on your evaluation set, your embeddings are not doing work.

Rerankers

A cross-encoder reranker (e.g., cross-encoder/ms-marco-MiniLM-L-6-v2) scores (query, passage) pairs jointly and is slower but much sharper than a bi-encoder. The standard shape:

1. Retrieve top-50 with a fast bi-encoder or BM25
2. Rerank to top-5 with the cross-encoder
3. Pass top-5 to the generator

Rerankers move the Pareto curve more than almost any other RAG change.

Evaluation — Two Loops, Not One

The single biggest RAG failure is measuring only the end-to-end answer. That hides whether a bad answer came from bad retrieval or bad generation.

Evaluate the two loops separately:

Retrieval evaluation (on a labeled (query, relevant_doc_ids) set):

• recall@k — did the correct passage appear in the top-k?
• MRR (mean reciprocal rank) — at what rank, on average, does the first correct passage appear?
• nDCG@k — graded relevance, if you have it

Generation evaluation (given the retrieved passages):

• faithfulness — does every claim in the answer appear in the retrieved passages? (LLM-as-judge with a strict rubric, or manual)
• answer relevance — does the answer address the question?
• citation correctness — does each cited passage actually support the claim it is attached to?

A matrix worth building once per RAG system:

retrieval correct	answer correct	diagnosis
✓	✓	pipeline working
✓	✗	generator ignored context — prompt or model issue
✗	✓	model hallucinated a correct answer — lucky, not reliable
✗	✗	retriever miss — fix the index first

The third row is the most dangerous: the answer looks correct but the retrieval is broken. Without separate evaluation you will ship this bug.

What To Inspect

• retrieval quality on held-out queries — keep a small evaluation set (50–200 query/passage pairs) and run it every time you change the chunker, embedder, or index
• failure clusters — group failed queries by type: entity queries, numeric queries, multi-hop queries; each fails differently
• context length vs. answer quality — does answer quality keep improving past top-3, or does extra context hurt (the "lost-in-the-middle" effect on long contexts)
• chunk-boundary bugs — the answer exists in the corpus but spans two chunks; the retriever never sees it intact
• stale index vs. live corpus — track when the index was last rebuilt and whether queries are asking about newer material

Failure Pattern

The classic failure is evaluating only end-to-end. A project ships with 68% end-to-end accuracy. Six months later someone discovers 40% of "correct" answers came from retrieval misses that the LLM happened to answer correctly from parametric knowledge. As the corpus grows, the luck runs out, and the system degrades without a diagnosable cause.

The second classic failure is over-engineering the orchestration. Teams wire a sophisticated framework before they have an evaluation set. The framework makes changes fast but the student cannot tell if any change helped. Evaluation first, scaffolding second.

Common Mistakes

• chunking too aggressively ("chunk_size=128") — answers get fragmented
• chunking too coarsely ("chunk_size=2048") — retrieval scores are dominated by irrelevant content in the same chunk
• using dense retrieval for queries dominated by exact entities/IDs (where BM25 wins)
• skipping the BM25 baseline and shipping dense without evidence it helps
• not storing (source, offset) with chunks — breaks citations and auditability
• no reranker when top-5 is the generator's actual context
• re-embedding the entire corpus every query (it should be indexed once, searched many times)
• passing more context than the model uses well (lost-in-the-middle)
• evaluating generation on gold-context answers while retrieval quietly degrades

Decision: Prompt-Only vs. RAG vs. Fine-Tune

A practical three-way:

option	knowledge source	cost to update	best for
prompt-only	model parameters + prompt	free	stable tasks, world knowledge, short corpora fitting in-context
RAG	retrieved corpus passages	rebuild index (minutes to hours)	dynamic knowledge, audit trail, medium-to-large corpora
fine-tune	baked into parameters	retrain (hours to days, $$)	stable domain language, strong style control, latency-sensitive inference

Two rules that settle most arguments:

• if the task is about facts that change, start with RAG
• if the task is about how the model writes or reasons, start with fine-tune

Practice

1. Build the minimal example above on a corpus of 200–1000 of your own documents. Measure recall@5 on 20 queries you write yourself.
2. Replace the dense retriever with BM25 using rank_bm25. Report where each wins, and where they agree.
3. Add a cross-encoder reranker. Re-measure recall@5 and the faithfulness of the final answer. Pick which stage the gain came from.
4. Build the retrieval/answer correctness matrix described above on 50 queries. Find at least one example of row 3 (lucky hallucination).
5. Vary chunk size across {128, 512, 1024} tokens with 10% overlap. Plot recall@5 for each. Defend one setting.
6. Write an inspection cell that, for a failing query, prints the top-10 retrieved passages with scores and the generator's answer. You will reuse this cell every time RAG breaks.

Runnable Example

A runnable RAG sketch using sentence-transformers + FAISS runs locally, not in the browser runner. Install with: pip install sentence-transformers faiss-cpu rank_bm25. The snippet in "Minimal Example" above is self-contained.

Longer Connection

RAG is an integration point, not a model. The retrieval half is classical ML — it belongs in the same mental slot as Leakage Patterns, Calibration and Thresholds, and Evaluation Metrics Deep Dive. The generation half lives next to Text Generation and Language Models and Attention and Transformers.

Two adjacent topics to read next:

• Text Generation and Language Models — decoding, prompt contract, tool-use; the generator side of RAG
• Transfer and Fine-Tuning — when fine-tuning is the right answer and RAG is not

For evaluation rigor, Selective Prediction and Review Budgets is the right frame: a RAG system that cannot cite should abstain, not guess.