A food delivery app wants to predict delivery time from distance. Here are 30 orders from one restaurant at normal hours. Find the best straight line through them by hand, see why it is the best, then watch the computer find it by itself. The formulas on the left change as you move the sliders.
Then build it in Python: the worked exampleEvery point is one order. Move the slope and the intercept until the line runs through the middle of the points. Each red line is an error: how far the line's guess is from the real time.
ŷ = m · x + c = 1.00 · x + 20.0
MSE = 1n Σ (yi − ŷi)2 = 130 × 1,442.8 = 48.1
RMSE = √MSE = 6.9 min (the line is about this many minutes off)
Lowest possible MSE: 5.4
Each pink square is one error multiplied by itself. MSE is the average area of these squares, so the best line is the one with the smallest squares. Big misses make very big squares, which is why MSE punishes them hard. In scikit-learn, LinearRegression().fit(X, y) finds coef_ ≈ 2.86 and intercept_ ≈ 9.00 for this data.
Every possible line has its own MSE. Put all of them on one 3D chart, with m and c on the floor and the MSE as the height, and you get a bowl. Your line is the red dot. The best line is the lowest point of the bowl.
Your line: MSE = 48.1
Bottom of the bowl: m = 2.86, c = 9.00, MSE = 5.4
Look at the shape: the bowl is a long, narrow valley. Raising the slope and lowering the intercept together keeps the error almost the same, because both can explain the same points. Keep this in mind for the next step.
Training a model means finding the bottom of this bowl. With one input there is a formula for it; with many inputs and bigger models, we walk down to it step by step.
Instead of a formula, start anywhere and keep stepping downhill. At each step, measure the slope of the bowl where you stand, then move a little in the downhill direction. The learning rate is how big each step is. It is a hyperparameter: you choose it, the model does not learn it.
m ← m − α · ∂MSE∂m, ∂MSE∂m = −2n Σ xi(yi − ŷi) = -67.40
c ← c − α · ∂MSE∂c, ∂MSE∂c = −2n Σ (yi − ŷi) = -3.60
Next step: m changes by 0.674, c by 0.036
Press run to start from where your line is now.
Try a learning rate of 0.020 or more: the steps overshoot the valley and the error explodes. Try 0.002: it is safe, but far too slow. Notice how the path first drops fast into the valley, then crawls along it. That narrow valley is why we scale features (Phase 2). In scikit-learn: SGDRegressor(learning_rate="constant", eta0=0.010, max_iter=1000).