Many valuable business problems are imbalanced: most transactions are genuine, most customers do not churn this month, most parts are not defective. Standard training and evaluation can go badly wrong when the class you care about is rare.
Why accuracy misleads
If a small share of cases are positive, a model that predicts "negative" for everything scores very high accuracy and is completely useless. Always look at the confusion matrix and class-specific metrics.
Better metrics
- Recall for the rare class: how many positives are caught.
- Precision: how many flagged cases are real.
- F1: balance of the two.
- PR-AUC (average precision): threshold-independent and focused on positives.
- Business metrics: cost saved, cases caught per 100 reviews.
Step 1: stratified splits
from sklearn.model_selection import train_test_split, StratifiedKFold
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)Stratifying keeps the share of positives the same in every split.
Step 2: class weights
Tell the model that mistakes on the rare class cost more:
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
LogisticRegression(class_weight="balanced", max_iter=1000)
RandomForestClassifier(class_weight="balanced", n_estimators=300)This is often the simplest and most effective first step.
Step 3: resampling (use with care)
- Undersampling the majority class: faster training, but throws away data.
- Oversampling the minority class: duplicates rare examples.
- Synthetic oversampling (SMOTE) creates new rare examples between existing ones (in the imbalanced-learn library).
Only resample the training data, inside cross-validation. Resampling before splitting leaks information and produces over-optimistic scores.
Step 4: tune the threshold
The default 0.5 threshold is rarely right for imbalanced problems. Use predicted probabilities and choose the threshold that matches business needs: a target recall, a fixed review capacity, or the best expected value.
proba = model.predict_proba(X_test)[:, 1]
flag = proba >= 0.25Step 5: better features beat clever tricks
Often the biggest gains come from features that separate the rare class: recent behaviour changes, ratios, history counts. Spend time here before reaching for complex resampling.
A checklist
- Measure class balance.
- Stratify splits.
- Choose metrics that reflect the rare class.
- Try class weights.
- Tune the threshold on validation data.
- Consider resampling only inside cross-validation.
- Report results in business terms.
Practise
Read the confusion matrix and ROC/AUC guides, then apply them in the hospital readmissions set, where the patients you most need to find are the minority.
Practise what you just read
Reading helps; writing the code yourself is what sticks. Start here: