K-Nearest Neighbours is the most intuitive machine learning algorithm: to predict something about a new case, find the most similar past cases and copy what happened to them.
How it works
- Store all training examples (KNN does no real "training").
- For a new point, measure its distance to every training point.
- Pick the K closest ones.
- Classification: take a majority vote of their labels. Regression: average their values.
Example: a fictional broadband company, LinkLane, wants to suggest a plan to a new customer. Find the five existing customers most similar in usage, devices and budget, and suggest the plan most of them chose.
Scaling is essential
KNN relies on distance. If one feature is monthly spend in rupees (hundreds or thousands) and another is number of devices (1 to 6), spend dominates every distance and devices barely matter. Always scale:
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
knn = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5))
knn.fit(X_train, y_train)
knn.score(X_test, y_test)Choosing K
- Small K (like 1): very sensitive to noise, jagged boundaries, overfitting.
- Large K: smoother, but may blur real differences and favour the majority class.
Try several values with cross-validation and pick the best. Odd numbers avoid ties in two-class problems.
from sklearn.model_selection import GridSearchCV
search = GridSearchCV(knn, {"kneighborsclassifier__n_neighbors": range(1, 31, 2)}, cv=5)
search.fit(X_train, y_train)
search.best_params_Distance measures
- Euclidean: straight-line distance; the default.
- Manhattan: sum of absolute differences; sometimes more robust.
- Weighted voting:
weights="distance"gives closer neighbours more say.
Strengths
- Simple to understand and explain: "customers like you chose this".
- No assumptions about the shape of the boundary.
- Works for classification and regression.
Weaknesses
- Slow predictions on large datasets, since it compares with every stored point.
- Curse of dimensionality: with many features, distances become less meaningful.
- Sensitive to irrelevant features and to scale.
- Needs the whole training set at prediction time.
When to use it
Small to medium datasets, a moderate number of meaningful features, and when "similar past cases" is a natural explanation, such as recommendations or simple similarity search.
See it in action
Drag points and change K in the KNN playground to watch the decision boundary change. Then follow the worked example in the KNN lesson and build a recommender in the gym programme practice set.
Practise what you just read
Reading helps; writing the code yourself is what sticks. Start here: