Skip to content

Track 19

RAG And Prompting Workflows

This track trains the prompt-vs-retrieval-vs-fine-tune decision with an offline course-support corpus. It forces a two-loop evaluation — retrieval recall first, answer correctness second — so that every gain is attributed to the correct half of the system.

Open RAG Topic Open Prompting Topic Use The Study Plan

Primary Goal

Pick The Right Lever

Use this track when you need to decide between prompting, retrieval, and fine-tuning with recall-level and answer-level evidence rather than by reputation.

You Will Practice

Two-Loop Evaluation

Closed-book baseline, frozen retrieval, prompt-style rerank, contrastive fine-tuned retrieval, recall@k curves, and a failure decomposition that separates retrieval misses from answer misses.

Exit Rule

Defend The Lever, Not The Score

You are done when you can name which half of the pipeline is responsible for the residual failures — and which lever you would invest in next.

Use This Track When

  • a model-over-knowledge-base task has a persistent error class and three plausible fixes
  • a headline accuracy number is hiding whether the retriever or the answer selector is the real bottleneck
  • you want retrieval, prompting, and fine-tuning to feel like three levers with different cost and reversibility profiles, not three synonyms

What This Track Is Training

This track trains one practical rule:

  • attribute every gain to retrieval or to rerank before reaching for fine-tuning

That means the learner should be able to keep these explicit:

  • what the closed-book baseline can already do — and whether that reveals a query-side leak
  • whether the top-1 passage is the correct one, separately from whether the answer is correct
  • whether rerank earned its cost by converting recall@k hits into recall@1 hits
  • whether a fine-tuned retriever beat the frozen retriever by more than per-seed variance

First Session

Use this order:

  1. Retrieval-Augmented Generation
  2. Prompting and Tool Use
  3. Clinic: Prompt Vs Retrieval Vs Fine-Tune — write the decision cold before running anything
  4. run academy/.venv/bin/python academy/labs/rag-and-prompting-workflows/src/rag_prompting_workflow.py
  5. write one sentence on which failure cell — hit_wrong, miss_lucky, or miss_wrong — is the biggest in your run

Full Track Loop

For the complete workflow:

  1. review the RAG and prompting topic pages
  2. do the matching clinic and keep the decision note
  3. run academy/.venv/bin/python academy/labs/rag-and-prompting-workflows/src/rag_prompting_workflow.py
  4. inspect rag_strategy_summary.csv first — do not jump to the plots
  5. open retrieval_failure_breakdown.csv and count the four cells per strategy
  6. open retrieval_recall_curves.png — the recall@1 vs. recall@5 gap tells you whether rerank has headroom
  7. open answer_correctness_by_strategy.png — pair the winner against its own recall, not against the other strategies only
  8. write the two-loop decision note: which lever you would invest in next, and why

What To Inspect

By the end of the track, the learner should have inspected:

  • one closed-book baseline that is honestly unable to answer out-of-context questions (and the cases where it is right, which expose a query-side leak)
  • one frozen retrieval run with recall@1, recall@3, and recall@5
  • one prompt-rerank pass that converts some top-5 hits into top-1 hits
  • one fine-tuned retriever compared against the frozen retriever by more than a single-point margin
  • one hit_wrong case where retrieval found the right passage and the answer selection still failed
  • one miss_lucky case where retrieval was wrong and the answer happened to land correctly — that is the cell that inflates headline numbers

Common Failure Modes

  • celebrating answer-correctness without looking at retrieval-correctness
  • declaring the fine-tuned retriever a winner on a one-point margin
  • comparing strategies at top-1 only, ignoring the recall@k curve
  • missing the miss_lucky cell and letting lucky retrieval wins inflate the summary
  • running the lab, looking at the final plot, and not writing the decision note

Exit Standard

Before leaving this track, the learner should be able to:

  • name which half of the pipeline — retrieval or answer selection — is responsible for the residual failures
  • defend either frozen_retriever, prompt_rerank, or finetuned_retriever as the next-week lever with recall and rerank-earn evidence
  • describe what would need to be measured on a held-out 40-query set before declaring the fine-tuned retriever production-ready
  • state the condition under which you would abandon retrieval entirely and invest in prompting or fine-tuning the answer model instead

That is enough to move into harder real-world RAG systems without letting a headline accuracy number pick the lever for you.