Clinic 04

Overfit Or Underfit?

The training curve is in front of you. The model is not performing well, but the reason matters more than the score. Diagnose the regime before you choose the fix.

Back To Clinics Open Learning Curves Topic Open The Full Track

Situation

Two Models, Two Curves

One model fits training perfectly but collapses on validation. The other never fits training well. The fix for each is opposite.

Your Job

Diagnose Then Fix

Name the regime for each model, choose the right intervention, and reject the wrong one.

Bad Habit To Avoid

More Data Fixes Everything

If your answer is always "get more data" regardless of the curve shape, you missed the diagnosis.

Situation

You are comparing two models on a tabular classification task. Both underperform, but for different reasons.

The packet says:

• Model A has near-perfect training accuracy but poor validation accuracy
• Model B has moderate training accuracy and nearly identical validation accuracy
• you have a fixed data budget and can only try one intervention per model

Artifact Packet

Read this packet before you decide:

model	train accuracy	val accuracy	train-val gap	complexity
deep_tree	0.993	0.724	0.269	max_depth=None, 1847 leaves
shallow_linear	0.681	0.672	0.009	logistic regression, C=0.01
dummy_majority	0.583	0.579	0.004	predicts majority class

Training curve summary (accuracy by training set fraction):

fraction	deep_tree train	deep_tree val	shallow_linear train	shallow_linear val
0.2	1.000	0.651	0.668	0.644
0.4	0.998	0.693	0.673	0.658
0.6	0.996	0.710	0.677	0.665
0.8	0.994	0.718	0.680	0.669
1.0	0.993	0.724	0.681	0.672

Decision Prompt

Write the note before you open the reveal.

Your note should answer:

1. What regime is deep_tree in?
2. What regime is shallow_linear in?
3. What is the single best intervention for each?
4. Which intervention would make things worse if applied to the wrong model?

Keep the note short. Four to six sentences is enough.

Strong Reasoning Looks Like

• it identifies the large train-val gap as overfitting and the flat low curve as underfitting
• it names a complexity-reducing intervention for the overfitter (pruning, regularization, fewer features)
• it names a capacity-increasing intervention for the underfitter (more features, higher C, nonlinear model)
• it explains why adding more data helps the overfitter but barely moves the underfitter
• it rejects "more data" as a universal fix

Common Wrong Moves

• calling both models "bad" without distinguishing the regimes
• applying regularization to the underfitting model
• adding complexity to the overfitting model
• ignoring the dummy baseline when evaluating the shallow model
• saying "try both interventions on both models" instead of committing

Run The Clinic In Browser

Validate Your Decision In Browser

Reference Reveal

Open only after you write the note

The reference diagnosis is: - `deep_tree` is **overfitting**: near-perfect training, large gap, validation improves slowly with more data - `shallow_linear` is **underfitting**: training and validation are close but both low, flat learning curve The reference interventions: - `deep_tree`: reduce complexity (prune, limit depth, add regularization) - `shallow_linear`: increase capacity (add polynomial features, raise C, switch to a nonlinear model) Why: - regularization on an underfitting model makes it worse by further restricting what it can learn - adding capacity to an overfitting model makes it worse by giving it more room to memorize - more data helps the overfitter converge but barely moves the underfitter because the model cannot represent the pattern regardless of sample size The practical lesson: the curve shape tells you the regime, and the regime tells you the direction of the fix.

What To Do Next

After this clinic:

1. open Learning Curves and Bias-Variance
2. run the matching learning curve example
3. use scikit-learn Validation and Tuning for the full tuning workflow