Every machine learning model faces the same tension: it must learn the real pattern in the data without memorising the noise. Too little learning is underfitting; too much is overfitting.
An everyday analogy
Two students prepare for an exam. One only skims the chapter titles (underfitting: too simple to answer anything). The other memorises last year's paper word for word (overfitting: perfect on that paper, lost when questions change). The student who understands the concepts does well on new questions. That is the goal for a model.
Underfitting
The model is too simple to capture the pattern. A straight line through clearly curved data is the classic example.
Signs: poor performance on training data and on test data.
Fixes: a more flexible model, better or more features, less regularization.
Overfitting
The model fits the training data too closely, including random noise. A very deep decision tree that creates a rule for almost every training row is the classic example.
Signs: excellent training score, much worse test score.
Fixes: a simpler model, more training data, regularization, limiting tree depth or minimum samples per leaf, fewer features, cross-validation for tuning.
Compare training and test scores
model.fit(X_train, y_train)
print("train:", model.score(X_train, y_train))
print("test: ", model.score(X_test, y_test))| Train | Test | Diagnosis |
|---|---|---|
| Low | Low | Underfitting |
| High | Much lower | Overfitting |
| High | Close to train | Good fit |
Regularization
Regularization adds a penalty for complexity to the model's training objective, so it prefers simpler solutions unless complexity really pays off.
Ridge (L2)
Penalises the squared size of coefficients. Coefficients shrink towards zero but rarely reach it. Good when many features each contribute a little.
Lasso (L1)
Penalises the absolute size of coefficients. Some coefficients become exactly zero, which removes those features: built-in feature selection.
from sklearn.linear_model import Ridge, Lasso
ridge = Ridge(alpha=1.0).fit(X_train, y_train)
lasso = Lasso(alpha=0.1).fit(X_train, y_train)alpha controls strength: higher means stronger penalty and a simpler model. Choose it with cross-validation, and scale features first, since the penalty treats all coefficients alike.
In other models
- Decision trees:
max_depth,min_samples_leaf. - Logistic regression in scikit-learn:
C(smaller C means stronger regularization). - Random forests reduce overfitting by averaging many trees.
The bias-variance trade-off
Underfitting is high bias; overfitting is high variance. Increasing complexity lowers bias and raises variance. The best model sits where test error is lowest.
See it happen
Our regularization playground lets you increase model complexity and the penalty strength and watch the curve go from too simple, to just right, to wildly overfit. The decision tree lesson shows the same idea with tree depth.
Practise what you just read
Reading helps; writing the code yourself is what sticks. Start here: