A model's score on the data it learned from tells you almost nothing about how it will perform on new data. Honest evaluation means testing on data the model has never seen.
The train-test split
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)- Train the model on the training set.
- Measure it once on the test set.
random_statemakes the split reproducible.
Common splits hold back around 20 to 30% for testing, but the right amount depends on how much data you have.
Stratify for classification
If only a small share of rows are positive (fraud, churn), a random split might put very few positives in the test set. Stratifying keeps the class balance the same in both parts:
train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)Why one split is not enough
A single split can be lucky or unlucky. Cross-validation repeats the process:
- Split the training data into k folds (often 5).
- Train on k โ 1 folds, validate on the remaining fold.
- Rotate until every fold has been the validation fold once.
- Average the scores.
from sklearn.model_selection import cross_val_score
scores = cross_val_score(model, X_train, y_train, cv=5, scoring="roc_auc")
scores.mean(), scores.std()The standard deviation tells you how stable the model is.
Train, validation and test
When you tune settings (tree depth, regularization strength), you are choosing based on validation scores, which slightly "uses up" that data. Keep a final test set aside and touch it only once, at the end, for the honest final number. Cross-validation on the training data plays the role of the validation set.
from sklearn.model_selection import GridSearchCV
search = GridSearchCV(model, {"max_depth": [3, 5, 8]}, cv=5).fit(X_train, y_train)
search.best_estimator_.score(X_test, y_test) # final check, onceTime-based data needs time-based splits
For forecasting or anything that changes over time, a random split lets the model "see the future". Train on earlier periods and test on later ones (TimeSeriesSplit in scikit-learn).
Data leakage: the silent score inflator
Leakage happens when information unavailable at prediction time sneaks into training:
- Preprocessing before splitting: scaling or imputing with statistics from the whole dataset. Fix: use a
Pipelineso preprocessing is fitted on training folds only. - Target leakage: a feature recorded after the outcome, such as "refund issued" when predicting returns.
- Duplicate entities across splits: the same customer appearing in both train and test.
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
cross_val_score(pipe, X_train, y_train, cv=5)If a score looks too good to be true, check for leakage before celebrating.
Practise
Every ML practice set on JobPrax, such as used car prices and trial users who pay, starts with a proper split, so you build the habit from the first model. Choose the right score with the metrics page.
Practise what you just read
Reading helps; writing the code yourself is what sticks. Start here: