Skip to content

Self-Supervised and Representation Learning

What This Is

Self-supervised learning (SSL) is the practice of training a model on a pretext task whose labels are generated from the data itself, and then reusing the learned representation on a downstream task. The representation is the product — the pretext task is only a device to get there.

Two families dominate practice:

  • masked modeling — hide part of an input and predict it (BERT's masked language modeling, MAE's masked image autoencoding, Wav2Vec's masked audio prediction)
  • contrastive learning — construct "positive" pairs of views of the same example and "negative" pairs of different examples, and train the encoder so positives are close in embedding space and negatives are far (SimCLR, MoCo, CLIP as a cross-modal cousin)

The thing a student must internalize: SSL is how modern encoders are pretrained. Fine-tuning, transfer learning, PEFT, and CLIP-style multimodal models all assume an SSL stage at the bottom of the stack.

When You Use It

  • you have a large unlabeled corpus and a small labeled corpus
  • you want an encoder that transfers across many downstream tasks rather than one specialist
  • the cost of labeling is the bottleneck; the cost of compute is not
  • you need robust features — SSL-trained encoders usually generalize better under distribution shift than task-specific ones

Do Not Use It When

  • you have enough labels that a plain supervised model trained end-to-end would outperform a frozen SSL encoder on your task
  • your data is too small for SSL (contrastive needs tens to hundreds of thousands of examples to stabilize)
  • your task is narrow and the generic representation is not the bottleneck

The Two Families At A Glance

family pretext task needed structure canonical examples
masked modeling predict hidden tokens / patches / frames a natural "units" structure (words, patches, frames) BERT, MAE, SimMIM, Wav2Vec 2.0
contrastive pull positive pairs together, push negatives apart a way to make two views of the same example SimCLR, MoCo, DINO, CLIP (cross-modal)
hybrid / non-contrastive bootstrap targets from a momentum encoder without negatives similar to contrastive BYOL, DINO, SimSiam

In practice, text is dominated by masked modeling; vision has swung between contrastive (2019–2022) and masked (MAE, 2022+); audio uses both.

Contrastive Learning — The Core Idea

Given an encoder f and two views x_i, x_i' of the same example (augmentations), the InfoNCE loss pulls positives together and pushes negatives apart:

L_i = -log [ exp(sim(z_i, z_i') / τ)  /  Σ_{j ≠ i} exp(sim(z_i, z_j) / τ) ]

where sim is cosine similarity, z = g(f(x)) is the projected embedding, and τ is a temperature. The thing that makes the loss work:

  • a strong augmentation pipeline — the two views of the same image must be visibly different; otherwise the encoder can pass a trivial shortcut
  • enough negatives — in-batch negatives, or a memory bank (MoCo), or a queue
  • a projection head g trained only during SSL and discarded at downstream time — this is the subtle lesson from SimCLR

Minimal Contrastive Setup (PyTorch Sketch)

Not a production recipe — the smallest code that preserves the shape of the loss:

import torch
import torch.nn.functional as F
from torch import nn

class ProjectionHead(nn.Module):
    def __init__(self, dim_in, dim_hidden=512, dim_out=128):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(dim_in, dim_hidden), nn.ReLU(),
            nn.Linear(dim_hidden, dim_out),
        )
    def forward(self, h):
        return F.normalize(self.net(h), dim=-1)

def info_nce(z1, z2, temperature=0.1):
    # z1, z2: (B, D) — paired views of B examples
    B = z1.size(0)
    z = torch.cat([z1, z2], dim=0)            # (2B, D)
    sim = z @ z.T / temperature               # (2B, 2B)
    # mask out self-pairs
    mask = torch.eye(2 * B, dtype=torch.bool, device=z.device)
    sim = sim.masked_fill(mask, float("-inf"))
    # positive targets: view-1 ↔ view-2
    targets = torch.arange(2 * B, device=z.device)
    targets = (targets + B) % (2 * B)
    return F.cross_entropy(sim, targets)

Four inspections you should run the first time:

  • loss at step 0 should be log(2B - 1) — random chance across the negatives
  • after a few hundred steps, pairs of views should be each other's nearest neighbour in the batch
  • augmentations should be strong enough that the model cannot identify an image from pixel statistics alone
  • the projection head should reduce nn.Linear features (e.g., 2048 dim) to 128 dim — keeping the encoder general while the projection absorbs pretext-specific geometry

Masked Modeling — The Core Idea

For masked language modeling (MLM, BERT-style): randomly mask ~15% of tokens, run the model, predict the masked tokens with cross-entropy. The model must learn a contextual representation because the only way to fill a hole is to read the rest of the sentence.

For masked image modeling (MAE-style): partition the image into patches, mask a high fraction (~75%), pass only the visible patches to a ViT encoder, and let a small decoder reconstruct the masked patches from the encoder output + learnable mask tokens. Two differences from MLM: masking ratio is much higher (images are more redundant) and the decoder is deliberately lightweight because it is discarded at downstream time.

Two inspections that matter:

  • reconstruction loss is a training signal, not a quality signal. A small reconstruction loss does not guarantee a useful representation.
  • linear probing (see below) is the real quality signal.

Evaluating A Representation

SSL is a two-stage workflow, and each stage needs its own evaluation.

Pretext evaluation (during SSL training):

  • InfoNCE loss / MLM perplexity — sanity check on training
  • alignment and uniformity diagnostics on the contrastive embeddings (Wang & Isola) — useful for contrastive
  • reconstruction loss on held-out data for masked modeling

These tell you the pretext task is learning. They do not tell you the representation transfers.

Downstream evaluation (the real test):

  • linear probing — freeze the encoder, train a single linear classifier on top, measure downstream accuracy. This is the canonical SSL benchmark.
  • k-NN evaluation — classify by nearest neighbour in embedding space using downstream labels. Fast, no training step.
  • full fine-tune — unfreeze and train end-to-end. Gives the ceiling; masks representation quality with model capacity.
  • few-shot transfer — measure accuracy with 1/5/10/100 labels per class. This is often where SSL encoders shine against supervised ones.

A production rule: when comparing two SSL recipes, always report linear probe and k-NN. Linear probe rewards separable features; k-NN rewards locally separable features. A recipe that wins one and loses the other is an important signal.

CLIP As A Cross-Modal Contrastive

CLIP is the same InfoNCE objective, but positives are (image, paired_caption) and negatives are (image, any_other_caption_in_the_batch). Two consequences:

  • the image encoder and text encoder are aligned — you can search images by text without a downstream classifier
  • zero-shot classification becomes "embed each class name and pick the nearest class" — a transformation of the image-classification problem

The lesson for this topic: contrastive scales naturally from single-modal (SimCLR) to multi-modal (CLIP) by swapping what counts as a positive pair.

What To Inspect

  • augmentation strength — in contrastive, too weak ⇒ shortcut learning; too strong ⇒ positives stop being the same thing
  • temperature τ — 0.07 for CLIP, 0.1–0.5 for SimCLR; lower τ sharpens the loss and increases effective difficulty
  • batch size / negative count — contrastive loves big batches because they provide more negatives; MoCo uses a queue to decouple batch size from negative count
  • collapse signatures — all embeddings shrink to the same vector (loss looks fine, embeddings are degenerate). Check the rank or std of embeddings; both should be high.
  • learning-rate warmup — critical for contrastive; skipping it is a leading cause of collapse in the first 1000 steps
  • what you measure at eval time — always report linear probe accuracy, not just pretext loss

Failure Pattern

The textbook failure is representation collapse: the encoder maps every input to the same embedding, the contrastive loss becomes small, and the downstream evaluation reveals the representation is useless. It looks like success until the downstream eval runs.

Three diagnostics that catch it:

  • embedding standard deviation across the batch is near zero
  • batchwise similarity matrix is uniformly high
  • linear probe accuracy is at chance level despite smooth pretext loss curves

The fix is usually some combination of: stronger augmentations, higher LR warmup, or the non-contrastive family (BYOL/DINO/SimSiam) which designs around collapse with momentum encoders and stop-gradients.

Common Mistakes

  • evaluating only pretext loss — ignoring downstream transfer
  • picking a too-weak augmentation set (random crop + flip without color jitter, for example)
  • keeping the projection head at downstream time instead of using the encoder features
  • linear probe with an overfit linear classifier (add LR tuning; very small L2 helps)
  • comparing across papers without matching the benchmark protocol (frozen vs. fine-tuned, batch size, schedule)
  • running masked modeling with a short context window — the model memorizes local statistics
  • reporting only fine-tune accuracy — it hides weak representations under model capacity

Decision Inside The Academy

You almost never train an SSL encoder from scratch. You almost always use one. A concrete decision tree:

  1. is there a pretrained encoder for your domain? Start there. Examples: BERT / RoBERTa (text), ResNet / ViT / DINOv2 (image), wav2vec 2.0 / HuBERT (audio), CLIP (joint image-text).
  2. is frozen good enough? Try linear probe first. If it hits the target, ship with a frozen encoder + simple head. See Transfer and Fine-Tuning.
  3. does partial unfreeze / LoRA close the gap? See PEFT and LoRA.
  4. do you need to fine-tune deeply? Full fine-tune or a new pretraining run; both are expensive and almost never the right first step.

Practice

  1. Take a 10k-image subset of CIFAR-10. Train SimCLR with ResNet-18 for a short budget (1k steps). Report linear probe accuracy vs. a random-init baseline.
  2. Collapse the augmentation to "flip only." Retrain. Confirm the linear probe degrades severely — this is the augmentation-strength lesson.
  3. Measure the embedding standard deviation every 100 steps during training. Verify it stays well above zero; run one deliberate misconfiguration (e.g., tiny LR with no warmup) and watch it collapse.
  4. Compare linear probe vs. k-NN on the same trained encoder. Note where they agree and where they diverge.
  5. Load a pretrained CLIP and compute the zero-shot accuracy on a 10-class image dataset. Report the gap between zero-shot and a 1-shot linear probe.
  6. Mask 15% of tokens in a small text corpus (Brown or WikiText-103 subset). Train a tiny transformer with MLM. Compare its linear-probe score on a downstream task against a random-init tiny transformer. Quantify the gap.

Runnable Example

Contrastive and masked modeling both require GPU-scale training to show a real effect, so the browser runner is not the right home for them. Work locally with pip install torch torchvision or pip install transformers. The InfoNCE snippet above runs on CPU — use it to verify shapes and the step-0 loss on synthetic data before scaling up.

Longer Connection

Self-supervised learning is the pretraining half of the transfer-learning story the academy tells later:

SSL is also where Beyond The Academy's self-supervised and VLM pointers now resolve to a real topic instead of an external link.