Semi-Supervised Learning

Labeled examples plus a potentially useful pool of unlabeled examples define a semi-supervised regime. These methods are worth evaluating when the unlabeled distribution and task assumptions are suitable; they are not guaranteed to beat a supervised baseline.

What This Is

You have n_L labeled rows and n_U unlabeled rows; n_U may be much larger than n_L, but that is not required. Semi-supervised methods use the unlabeled pool as a second source of signal — by pseudo-labeling it, shaping the decision surface, or pretraining a representation — and compare the result with a labeled-only baseline.

The three families you should know by name:

• Self-training (pseudo-labeling). Train on labeled data. Predict on unlabeled. Keep predictions whose confidence clears a threshold. Add those pseudo-labeled rows to the training set. Retrain. Iterate until nothing new is added or quality on a held-out labeled set starts dropping.
• Label propagation / label spreading. Build a similarity graph over all points (labeled + unlabeled). Let labels flow along the graph edges until stable. The unlabeled points inherit labels from their neighborhood structure. Works well when the manifold assumption holds (points close in feature space share a label).
• Consistency regularization. Push the model to produce similar outputs on two perturbed versions of the same unlabeled input. Families include FixMatch, MixMatch, Π-model. Stronger than self-training when augmentations are meaningful.

A fourth move that is not strictly SSL but often beats it: train a self-supervised representation on all the unlabeled data, then fit a linear head on the small labeled set. See Self-Supervised and Representation Learning.

When You Use It

• labeled data is expensive or slow to collect, unlabeled is cheap
• the supervised model plateaus fast and you suspect it is label-limited, not capacity-limited
• the unlabeled distribution matches the labeled one (same population, same feature definitions)

Do Not Use It When

• labels are cheap — just collect more
• the labeled and unlabeled pools come from different distributions (pseudo-labels will drift)
• your supervised model is already near its Bayes-error floor — SSL cannot invent signal that is not there
• you cannot hold out a clean labeled set for validation — without it you cannot tell if pseudo-labels are helping or hurting

Pseudo-Labeling In Practice

The core loop:

1. split labeled data into train / val
2. train model on train
3. predict on unlabeled; keep rows where max(softmax) >= τ (often τ ≈ 0.9)
4. add kept rows with their predicted label to the training set
5. retrain from scratch (or warm-start)
6. evaluate on val; if val improved, iterate; if it dropped, stop and use the previous round's model

The threshold τ matters. Too low can inject label noise; too high can admit too few rows. Select a fixed, scheduled, or class-specific threshold using a clean labeled validation set. Do not lower it merely because another round has completed: confidence can be miscalibrated and differs by class.

Label Propagation In Practice

from sklearn.semi_supervised import LabelSpreading

clf = LabelSpreading(kernel="knn", n_neighbors=7, alpha=0.2)
# y: labeled = class id, unlabeled = -1
clf.fit(X, y)
y_pred = clf.transduction_  # per-row predicted label including previously unlabeled

The hard part is the graph. Euclidean distance on raw features often does not represent semantic similarity. Scaling, PCA, or a learned embedding usually helps.

What To Inspect

• labeled val accuracy vs round — should climb, then plateau, then drop if you overrun
• pseudo-label class distribution — compare it with labeled prevalence and plausible deployment prevalence; a large shift is a warning to investigate, not proof of drift
• pseudo-label confidence distribution — a big peak near the threshold means τ is near a cliff; shift it to see
• agreement between rounds — if round N flips many labels that round N-1 assigned, the model is not stabilizing
• held-out performance on the target distribution — keep the final test set untouched until selection; also inspect calibration and important slices when they affect deployment

Failure Pattern

• confirmation bias. The model is confident about wrong predictions; those get pseudo-labeled in; the model gets more confident about the same wrong thing. The pool poisons itself. Symptom: val accuracy climbs for a round or two then drops.
• class imbalance amplification. The majority class starts slightly over-represented in pseudo-labels; next round the minority classes shrink; eventually you have a majority-class-predictor that looks confident. Fix: per-class thresholds or balanced sampling.
• distribution shift. Unlabeled comes from a slightly different source (time, geography, sensor). Pseudo-labels push the model toward the new distribution, which hurts on the original labeled test.
• no validation signal. You ran 10 rounds and picked "the last one" without checking val. You have no evidence it is better than round 0.

Quick Checks

• before pseudo-labeling: is there a clean labeled val set that will NOT be contaminated?
• is the unlabeled pool from the same distribution as the labeled pool?
• is the initial supervised model well-calibrated enough for confidence thresholds to be meaningful?
• does each round of pseudo-labeling use a fresh model (trained from scratch) or warm-start? (either is fine, but know which)
• is τ fixed or annealed across rounds?

Practice

Run .venv/bin/python labs/semi-supervised-learning/src/ssl_workflow.py:

• it generates a synthetic 2D two-moons dataset with only 20 labeled points and 2000 unlabeled
• it compares four strategies: labeled-only logistic regression, self-training with τ=0.9, self-training with per-class balanced thresholds, and label spreading
• it reports accuracy on a held-out clean labeled set and the round-by-round val trajectory

After running it, you should be able to explain when pseudo-labeling helps, when it hurts, and how confirmation bias looks in the round-by-round curve.

Runnable Example

From the repository root, run:

.venv/bin/python labs/semi-supervised-learning/src/ssl_workflow.py

Compare the labeled-only baseline with each semi-supervised strategy, then inspect whether validation quality improves or compounds early pseudo-label mistakes.

Longer Connection

Semi-supervised learning is one answer to "labels are expensive." The others:

• active learning — ask a human to label the points the model is most uncertain about, not random ones
• weak supervision — write heuristic labeling functions over the unlabeled pool and denoise them
• self-supervised pretraining — train a representation on the unlabeled pool with no labels, then linear-probe

These are not mutually exclusive. A production setup often stacks self-supervised pretraining + pseudo-labeling + active-learning acquisition. The exam setting typically gives you one labeled pool and one unlabeled pool and asks you to do the best you can — in that case, start with pseudo-labeling on a well-calibrated model and a real held-out val set.