Skip to content

Language Modeling Fundamentals

What This Is

A language model assigns a probability to a sequence of tokens. In practice it is trained to predict the next token given the context, either in an autoregressive left-to-right scan (GPT-style) or by filling in masked positions (BERT-style).

The practical lesson is that almost every modern NLP system — chat models, translation models, summarizers, code completion, RAG answer models — is some training scheme built on top of next-token prediction. Understanding what a language model is actually optimized for (and is not) is the difference between using one as a tool and treating its outputs as truth.

When You Use It

  • pretraining a base model you will later fine-tune or prompt
  • evaluating a model intrinsically with perplexity
  • choosing between a masked (encoder-only) and an autoregressive (decoder-only) backbone for your task
  • framing a new task as a next-token prediction problem

Two Training Regimes

Autoregressive (decoder-only, "causal LM")

Model p(y_1, ..., y_T) by the chain rule:

p(y_1, ..., y_T) = Π_t p(y_t | y_1, ..., y_{t-1})

At each position the model sees only past tokens (enforced with a causal mask in self-attention). Loss is cross-entropy over every position.

This is the GPT family. Strengths: open-ended generation, conditional generation, seamless transfer to chat. Weakness: cannot use right-side context for representation.

Masked (encoder-only, "MLM")

Randomly mask some tokens in the input; the model must predict the masked tokens using both left and right context:

input:  the [MASK] sat on the mat
target: predict "cat" at the masked position

Loss is cross-entropy over masked positions only.

This is the BERT family. Strengths: strong contextual representations for classification, extraction, and retrieval; both-sides context. Weakness: cannot generate text end-to-end without an extra decoder.

The choice

Task Backbone
classification, NER, extraction, retrieval masked (BERT-style)
generation, chat, completion autoregressive (GPT-style)
encoder-decoder tasks (translation, summarization) both — encoder is masked, decoder is autoregressive

Teacher Forcing — The Training Ritual

During autoregressive training, the model is given the true previous tokens at every position (not its own predictions). This is teacher forcing. It is what makes training parallel across positions and stable.

The cost is exposure bias: at inference the model conditions on its own (possibly wrong) outputs, a distribution it has never seen. Scheduled sampling and minimum-risk training are the classic mitigations; at large scale the gap is usually accepted and handled with good decoding (see Encoder-Decoder And Machine Translation).

Perplexity — The Standard Intrinsic Metric

Perplexity is the exponential of the average negative log-likelihood per token:

PPL = exp( (1/N) Σ_i -log p(y_i | context_i) )

Interpretations:

  • PPL of 20 means the model is as uncertain as if it had 20 equally likely choices at every step
  • PPL is tokenizer-dependent — a sub-word tokenizer gives lower PPL than a character tokenizer on the same text, but that does not mean it "understands" more
  • PPL measures intrinsic likelihood, not downstream utility — a model with lower PPL may still perform worse on a task

Use perplexity for model comparison when the tokenizer is the same. For cross-tokenizer comparisons, use bits-per-byte or downstream task metrics instead.

Context Window And Position

Transformers have a finite context window — the maximum number of tokens they attend over. Everything outside is invisible.

Position information is injected separately from token embeddings, because self-attention is permutation-invariant without it:

Scheme Notes
absolute (learned) one learned embedding per position; fixed max length
sinusoidal fixed sin/cos; extrapolates poorly past training length
rotary (RoPE) rotation applied inside attention; modern default
ALiBi linear bias on attention scores; extrapolates well

Most production open-weights models use RoPE. Scaling context length beyond training length needs position-interpolation or extended-training tricks; it is not free.

Subword Tokenization — The Hidden Contract

A language model does not see words; it sees subword tokens. The tokenizer contract — usually BPE, WordPiece, or SentencePiece — decides what the model's vocabulary is and therefore what its perplexity curve even means.

See Tokenization Mechanics for the details. The practical point: changing the tokenizer invalidates prior loss numbers, and a mismatched tokenizer at inference (loaded wrong) is one of the most common silent bugs in LM workflows.

Generation At Inference

Once trained, an autoregressive model generates by sampling or searching over its next-token distribution. See Encoder-Decoder And Machine Translation for decoding strategies (greedy, beam, top-k, nucleus, temperature). The decoding strategy is not a training choice — it is an inference-time knob that substantially changes what the same model produces.

What To Inspect

  • the per-token loss curve during training — flat loss after a few steps signals the model is not learning
  • perplexity on a held-out set, with the same tokenizer as training
  • whether the data order and random masking are actually random (a common seed bug)
  • top-k predictions for a few prompts — do they look linguistically plausible
  • whether <EOS> gets produced at a reasonable rate for generation tasks
  • token length distribution vs. reference — systematic too-short or too-long answers reveal sampling bugs

Failure Pattern

Comparing two language models by perplexity across different tokenizers and declaring one "better." Different tokenizers give different PPL numbers on the same text; the comparison is meaningless without a shared tokenization or a per-byte normalization.

A second failure pattern is treating perplexity as a proxy for correctness. A model can be fluent and wrong; perplexity only measures how well it matches the distribution, not whether its outputs are useful.

A third failure pattern is loading the model with one tokenizer and the input with a different tokenizer — tokens silently mismatch and the model produces garbage without any explicit error.

A fourth failure pattern is prompting a masked (BERT-style) model as if it were autoregressive. It will not generate text; it will only infill masked positions.

Quick Checks

  1. Is the tokenizer loaded from the same checkpoint as the model?
  2. Does held-out perplexity track training perplexity, or has the model overfit?
  3. Are causal masks correctly applied in decoder-only training?
  4. For MLM training, is the masking rate reasonable (usually 15%)?
  5. Is the context window long enough for your task? If not, truncation will silently drop information.

Practice

  1. Train a small character-level autoregressive model on Tiny Shakespeare and measure perplexity.
  2. Train a small masked model on the same text with 15% masking. Compare the representations on a downstream classification task.
  3. Vary the mask rate from 5% to 40% and plot downstream task accuracy.
  4. Compute teacher-forced loss and free-running loss (sampled) on the same autoregressive model. Explain the gap.
  5. Replace a learned positional embedding with sinusoidal. Compare generation quality past the training length.
  6. Explain why perplexity with sub-word tokens is lower than with characters and why that does not imply understanding.
  7. Describe one task where masked modeling is the better backbone and one where autoregressive is.
  8. Explain the causal mask's role in decoder-only self-attention.
  9. Describe one reason a model might produce fluent but factually wrong text.
  10. State the relationship between tokenizer changes and prior perplexity numbers.

Longer Connection

Language modeling fundamentals sit next to:

Language modeling is next-token prediction, and almost every modern NLP system is downstream of that single training objective. Understanding its strengths and its silences is the prerequisite for everything else.