Skip to content

Loss and Gradients Intuition

What This Is

A loss function is a single number that says how wrong the model is right now. A gradient tells you which direction to nudge each parameter to make the loss smaller. Gradient descent is the loop that does that nudge over and over.

Almost every academy page past this one assumes you know what a loss is, what a gradient is, and why "training" means "follow the gradient downhill". This page builds that picture from scratch, without pretending the math is hard.

If you have already done some calculus, you can read this page quickly. If you have not, you can still follow it — the mental picture is what matters.

When You Use It

  • you saw loss.backward() in PyTorch and want to know what it is doing
  • you ran a scikit-learn .fit() and want to know what "fitting" actually means
  • you are about to pick a loss function for a task and do not want to guess
  • a training curve is misbehaving and you need the vocabulary to describe why

A Loss In One Line

A loss function takes a prediction and a truth and returns a number:

loss = compare(prediction, truth)

Larger = more wrong. Smaller = less wrong. Zero = perfect.

The entire goal of training is: make this number smaller on training data, in a way that also makes it smaller on data you have not seen.

The Three Losses You Will See First

Mean squared error (regression)

loss = ((predictions - targets) ** 2).mean()

Used when the target is a number (price, temperature, count). Penalty grows quadratically — being off by 10 is 100 times worse than being off by 1.

Binary cross-entropy (two-class classification)

# predictions are probabilities in (0, 1)
loss = -(targets * log(p) + (1 - targets) * log(1 - p)).mean()

Used when the target is 0 or 1. Rewards the model for putting high probability on the correct class and heavily punishes confident wrong answers.

Cross-entropy (multi-class classification)

# predictions are class probabilities that sum to 1
loss = -log(p_true_class).mean()

The natural extension of binary cross-entropy to more classes. This is what torch.nn.CrossEntropyLoss computes.

These three cover 90% of what you will see in the academy. Evaluation Metrics Deep Dive covers the evaluation-time metrics; the losses above are the ones used during training.

Why Not Just Use Accuracy As The Loss

Because accuracy is flat. If the model's probability for the correct class goes from 0.3 to 0.49, accuracy does not move — both are still wrong. Cross-entropy sees the change and rewards it. Training needs a smooth signal so the nudge step has something to push on.

Rule of thumb: train with a smooth loss, evaluate with the metric you actually care about.

Gradients — The Direction To Nudge

Think of the loss as a landscape. Each parameter is an axis. Each point in that landscape is a specific setting of all parameters. The height at that point is the loss.

Training is: start somewhere, look at the slope underfoot, take a step downhill, repeat.

The gradient is the vector pointing uphill. To go downhill, take a step in the opposite direction:

parameter ← parameter - learning_rate * gradient

That is gradient descent. Every optimizer in deep learning is a variation on this.

A Worked Toy Example

Predict y from x with a single-parameter model: ŷ = w * x. One training example: x = 2, y = 6.

Loss:

L(w) = (ŷ - y)^2 = (w * 2 - 6)^2 = (2w - 6)^2

Gradient (derivative of loss with respect to w):

dL/dw = 2 * (2w - 6) * 2 = 8w - 24

At w = 1: gradient is 8 * 1 - 24 = -16. Negative gradient means loss decreases if we increase w. Take a step:

w ← 1 - 0.1 * (-16) = 1 + 1.6 = 2.6

At w = 2.6: loss is (2 * 2.6 - 6)^2 = 0.64. We went from a loss of 16 down to 0.64 in one step. With a few more steps, w converges to exactly 3, which is the true slope.

This is the whole of gradient descent, condensed. Real models have millions of parameters and non-trivial loss surfaces, but every step is the same move: compute the gradient, multiply by a learning rate, subtract.

The Learning Rate

The size of the step. Too small and training takes forever. Too large and the step overshoots, loss goes up instead of down, and training diverges.

  • too small → training "stalls"; loss decreases painfully slowly
  • about right → loss decreases smoothly and keeps improving
  • too large → loss bounces or increases; training blows up

The learning rate is the single most important hyperparameter in deep learning. Optimizers and Regularization goes into how to pick it honestly.

Backpropagation — Gradient For Many Parameters At Once

In a model with millions of parameters, computing the gradient by hand is impossible. Backpropagation is the algorithm that computes all gradients efficiently by reusing intermediate results.

The key idea is the chain rule from calculus. For a model that is a sequence of layers:

x → layer_1 → h_1 → layer_2 → h_2 → layer_3 → ŷ → loss

The gradient of the loss with respect to any parameter can be computed by multiplying local gradients along the path from that parameter to the loss. PyTorch and every deep-learning library do this automatically.

In PyTorch, the whole thing is three lines:

loss = loss_fn(model(x), y)
loss.backward()        # fills in parameter.grad for every parameter
optimizer.step()       # takes a step in the negative-gradient direction

Stochastic Gradient Descent

Computing the gradient over every training example for every step is too expensive for large datasets. Instead, you compute it over a small random batch (usually 32 to 256 examples) and take a step. This is stochastic gradient descent.

The noise from using a batch instead of the whole dataset actually helps training — it lets the optimizer escape shallow minima.

Pseudocode:

for epoch in range(num_epochs):
    for batch in dataloader:
        loss = loss_fn(model(batch.x), batch.y)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

An epoch is one pass through the whole training dataset. A step is one gradient update.

Why Loss Landscapes Are Non-Convex

For linear regression and logistic regression, the loss has one bowl-shaped minimum — gradient descent will find it. For neural networks, the loss has many local minima, saddle points, and flat regions. This is called a non-convex loss.

Modern deep learning works anyway, for reasons that are partly mathematical and partly empirical:

  • most local minima are close in loss value to the global minimum
  • SGD noise helps escape saddle points
  • over-parameterized networks have paths of low loss connecting different minima

The practical consequence: you do not need to find the best possible weights, only good enough weights that generalize.

Overfitting As A Loss Story

Training loss keeps dropping. Validation loss stops dropping and starts climbing. That gap is overfitting — the model is memorizing training-set noise.

Regularizers (weight decay, dropout, augmentation) are tools that make it harder for the model to drop training loss without also dropping validation loss. That is why they matter. Optimizers and Regularization covers the toolbox.

Choosing A Loss For A New Task

Default recipe:

Task shape Loss
regression (predict a number) mean squared error or Huber
binary classification binary cross-entropy (also called log loss)
multi-class single-label cross-entropy
multi-class multi-label binary cross-entropy per label
imbalanced classification weighted cross-entropy or focal loss
ranking pairwise hinge, lambda-rank, or listwise loss
contrastive / similarity InfoNCE, triplet loss

Start from the default. Reach for a fancy loss only when the default does not match the task's costs. The loss should correspond to the evaluation metric you care about, not to what was popular in the last paper you read.

What To Inspect During Training

  • the loss curve over steps — is it smooth, bouncy, or flat?
  • training vs validation loss — is the gap widening?
  • gradient norms — are they huge (exploding) or near zero (vanishing)?
  • a handful of actual predictions — do they match the loss story?
  • the learning rate trajectory — is a scheduler behaving as intended?

PyTorch Training Loops and Debugging Deep Learning both lean heavily on reading these signals.

Failure Pattern

Confusing loss with metric. Cross-entropy loss is not accuracy. Mean squared error is not R². A model with lower loss on the validation set usually has a better metric too, but not always — especially when the loss and metric are misaligned (e.g., training with MSE but evaluating with median absolute error).

Another failure: watching only the training loss. Training loss always goes down. Validation loss is the honest signal.

A third: picking a loss because a tutorial used it, not because it matches the task's cost. A fraud-detection model that weights every class equally will produce a fluent, accurate-looking, useless classifier.

Common Mistakes

  • using accuracy as the training loss
  • forgetting to zero out gradients between steps (PyTorch's optimizer.zero_grad())
  • training with a loss that does not match the evaluation metric
  • setting the learning rate without any measurement
  • taking one wild epoch as evidence that training is stable
  • comparing models on different losses and acting surprised that the picture disagrees
  • treating gradient magnitudes as unrelated to the LR choice

Practice

  1. Compute the gradient of L(w) = (2w - 6)^2 by hand at w = 0, w = 3, w = 5. Confirm that it points toward w = 3.
  2. Train a linear regression by gradient descent in pure NumPy. Plot loss vs step.
  3. Try three learning rates: way too small, about right, way too large. Describe what each curve looks like.
  4. Train the same model with MSE loss and with mean absolute error. Explain which one is more sensitive to outliers and why.
  5. Plot training loss and validation loss together for a classifier. Mark the epoch where overfitting begins.
  6. Explain in one sentence why cross-entropy is a better training loss than accuracy.

Runnable Example

See examples/first-steps/first_model.py and inspect how the optimizer updates the model's weights.

Longer Connection

Continue with What Is A Model if the parameter-as-knob picture is still fuzzy, Probability and Linear Algebra Refresher for the math you will keep running into, and PyTorch Training Loops to see the loop above expressed in full code.