Text Generation and Language Models¶

What This Is¶

This page is about deciding when free-form generation is the right tool and how to use it without fooling yourself. The first question is usually not temperature or beam width. The first question is:

do I actually need generation, or would classification, retrieval, reranking, or template selection be safer

Generation is powerful, but it is also the easiest way to produce fluent nonsense. The second most common failure is reaching for a fine-tune when a tighter prompt or a small retrieval set would have solved the problem with less risk.

This topic covers four closely related decisions:

when to generate vs. classify vs. retrieve vs. fine-tune
how to set the decoding rule and defend it
how to evaluate open-ended outputs honestly
when to add retrieval (RAG) or tool use on top of a base model

When You Use It¶

summarization, answer drafting, rewriting, open-ended explanation
code completion or structured output that still needs natural-language content
constrained generation where the output itself must be text
chat, instruction following, or multi-turn interaction
deciding whether a problem needs a generative model at all

Start With The Task Decision¶

Choose the smallest workflow that fits the task:

Task shape	Better first move	Why
fixed label or category	classifier or reranker	you need a decision, not open text
answer from trusted documents	retrieval plus grounded answer (RAG)	facts matter more than surface fluency
fixed schema extraction	constrained output, parser, or function-calling	format stability matters more than creativity
explanation, draft, or rewrite	generation	the output itself must be language
multi-step task with external data	generation + tool use	the model needs to call APIs, not just write

Many teams reach for generation too early. That usually creates evaluation problems, not better products.

Prompt vs Retrieval vs Fine-Tune¶

The three-way decision that appears in almost every production LLM project:

Option	When it is the right call	Risks
Better prompt	the base model already knows the domain; output contract is the gap	brittle, hard to version
Retrieval (RAG)	the base model lacks facts or the facts change often	retrieval quality becomes the bottleneck
Fine-tune	format or style is reliably off, OR tokens saved per call matter at scale	cost, staleness, evaluation overhead

Decision ladder:

Can a sharper prompt with 2–3 in-context examples hit the target? If yes, stop.
Does the task involve facts that the model cannot reliably produce from weights? If yes, add retrieval before fine-tuning.
Is the cost or latency of a long prompt the bottleneck, or is the output format drifting even under good prompts? If yes, consider fine-tuning.

Fine-tuning is almost never the first move. It is the last move after prompting and retrieval have been exhausted.

The Generation Loop¶

Use the same loop every time:

define the output contract
choose a conservative decoding rule
inspect a small batch of outputs
verify facts, format, and refusal behavior
decide whether prompting is enough or whether you need stronger grounding or tuning

If you skip the output contract, debugging becomes much harder.

Minimal Pattern¶

This is the shortest useful generation pattern:

generated = model.generate(
    **inputs,
    max_new_tokens=120,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
)

Those knobs matter only after the task is defined clearly.

Decoding Rules¶

Greedy: do_sample=False. Deterministic, safest for factual tasks, most boring for creative tasks. Strong default when you want reproducibility.
Temperature sampling: do_sample=True, temperature=T. Low T (0.2–0.4) for factual; moderate T (0.7–0.9) for writing; high T (>1) almost never.
Top-k: only sample from the k most likely tokens. Cheap quality floor.
Top-p (nucleus): sample from the smallest set of tokens whose cumulative probability exceeds p. Adapts per step.
Beam search: keeps the top-n most likely sequences at each step. Strong for translation and tight-structure tasks; tends to be bland for open-ended writing.
Contrastive decoding / typical sampling: newer schemes that can reduce repetition without the blandness of beams.

Practical defaults:

factual Q&A: greedy or temperature 0.2
summarization: beam search width 4 or temperature 0.3 with top-p 0.9
creative writing: temperature 0.7–0.9, top-p 0.9
code completion: temperature 0.2, top-p 0.95

Tune decoding last, not first. A well-defined task and a good prompt matter more.

Retrieval-Augmented Generation (RAG)¶

When the base model lacks facts or the facts change, retrieve documents at query time and ground the generation in them.

Minimal RAG loop:

# indexing (done once)
docs = chunk(corpus, size=512, overlap=64)
embeddings = embed(docs)
index.add(embeddings)

# per query
q_emb = embed(query)
top_k = index.search(q_emb, k=5)
context = "\n\n".join(docs[i] for i in top_k)
prompt = f"Use only the following context to answer.\n\n{context}\n\nQuestion: {query}"
answer = generate(prompt)

The parts that usually go wrong:

chunking — too small and each chunk has no standalone meaning; too large and retrieval becomes noisy
embedding model — a weak embedder is the single biggest quality regression; evaluate retrieval before you evaluate generation
reranking — cross-encoder rerankers often double recall-at-5 on the top-k retrieved set
prompt contract — the instruction must explicitly tell the model to use only the retrieved context and to refuse when the context is insufficient

Evaluate RAG in two pieces:

retrieval: for each question, is the answer-bearing chunk in the top-k? (recall-at-k)
generation: given the right context, does the model produce a correct, grounded answer? (exact match, F1, or human score)

Mixing these two scores into one number hides which side is broken.

Prompting Patterns That Matter¶

Output contract — say in the prompt what the output must look like, in plain terms. If the format must be JSON, show a JSON example.
Few-shot — 2–5 worked examples in the prompt. More than 8 rarely helps and wastes tokens.
Chain-of-thought (CoT) — instruct the model to reason step by step. Helps on math and multi-step reasoning. Adds tokens, cost, and latency. Do not enable blindly.
System prompt — high-level rules that persist across turns. Keep it short; long system prompts are often ignored.
Structured output — if the task is really parsing, use function-calling or JSON-schema-constrained decoding rather than free text.

When CoT helps vs. hurts:

helps: arithmetic, multi-hop QA, logic puzzles
hurts: classification (adds noise), fluent writing (adds awkward justifications), latency-sensitive paths

Tool Use and Function Calling¶

When the model needs to do something it cannot do from weights alone (look up a date, run a calculation, query a database), expose tools:

tools = [
    {
        "name": "get_weather",
        "description": "Get the current weather for a city",
        "parameters": {"type": "object", "properties": {"city": {"type": "string"}}},
    },
]

response = model.chat(
    messages=[{"role": "user", "content": "What should I wear in Paris today?"}],
    tools=tools,
)

The model returns either a text answer or a tool call. Your code executes the tool, feeds the result back, and the model generates the final answer.

Evaluation for tool use:

tool-selection accuracy — did the model pick the right tool?
argument correctness — are the arguments well-formed and meaningful?
end-to-end success — does the final answer use the tool result correctly?

A model that picks the right tool but hallucinates arguments is useless.

Evaluation Honestly¶

Open-ended generation is hard to evaluate because "looks fine" is not a metric. Use a stack:

Exact-match / F1 — for extractive or structured tasks
ROUGE / BLEU — for summarization or translation, as a floor, not a ceiling. They reward surface overlap.
BERTScore / embedding similarity — slightly better for paraphrase
Reference-free scoring — reading a sample and counting hallucinations, format violations, refusal failures
LLM-as-judge — cheap but introduces its own bias; always pair with human spot-checks
Red-team slice — fixed list of adversarial, boundary, and failure prompts; regression-test every release on it

One metric is never enough. A good generation evaluation has at least one surface metric (ROUGE/BLEU/exact-match), one semantic metric (BERTScore or LLM-judge), and one human-checked slice.

Refusal and Safety Evaluation¶

Generation evaluation is incomplete without testing what the model refuses. Build a refusal set:

prompts the model should refuse (requests for harmful content)
prompts the model should not refuse (benign requests that sound risky)
prompts where the right answer is "I don't know"

Track:

false-refusal rate on the benign set
pass-through rate on the harmful set
calibration on the "I don't know" set

If you skip this, your fine-tune or prompt change can silently regress safety behavior while the main metric looks fine.

What To Inspect¶

one batch of actual outputs, not only scores
length distribution — are outputs suddenly much longer or shorter after a change?
format violation rate — for tasks with output contracts
refusal behavior on a fixed adversarial and benign set
retrieval top-k for RAG tasks before blaming the generator
tool-call logs for tool-using agents
a token-level log-prob plot for decoder sanity checks

Failure Modes¶

Each of these is worth a named drill:

polished hallucinations — fluent, wrong, confident. The hardest to catch without a reference.
vague answers that sound complete but do not commit to the claim
brittle prompt behavior hidden by a few cherry-picked examples
evaluation with only BLEU, ROUGE, or "it looks fine"
RAG systems where the generator is blamed for a retrieval failure
tool-use agents where the right tool is picked with wrong arguments
fine-tunes that fix the target metric and silently regress refusal or safety

Common Mistakes¶

using generation for tasks that are really classification or ranking
setting temperature too high on factual tasks
blaming the model for an ambiguous prompt with no output contract
evaluating only with surface metrics
fine-tuning on too little domain data and calling the result specialization
forgetting to test failure cases, refusals, and malformed outputs
skipping retrieval evaluation before generation evaluation in a RAG pipeline
enabling CoT on every task and paying for tokens that did not help
treating LLM-as-judge as ground truth

A Good Generation Note¶

After one run, the learner should be able to say:

why generation was chosen over a simpler workflow
what the output contract was
which decoding rule was used and why
what failure pattern appeared first
what evidence would justify grounding (RAG), tool use, or fine-tuning

Practice¶

Turn one generation task into a more constrained output contract.
Compare greedy decoding against a moderate sampling setup on the same prompt. Report the first real difference.
Build a minimal RAG pipeline with 5 retrieved chunks and evaluate retrieval and generation separately. Explain which side is the bottleneck.
Add chain-of-thought to a classification task and show whether it hurt.
Take a fluent, confident output and verify whether it is actually correct. Name the failure if it is not.
Build a refusal-eval set of 20 prompts and measure false-refusal vs pass-through rates.
Give a model a tool and evaluate tool-selection accuracy vs. argument correctness separately.
Defend one choice: prompt engineering, RAG, or fine-tuning, for a specific fixed task, using the ladder above.

Runnable Example¶

Longer Connection¶

Continue with Attention and Transformers for the architecture behind modern language models, Text Representations and Order for the simpler text baselines that should come first, and Text Workflows Beyond Classification for fuller retrieval and generation-style workflows.