A classifier that outputs probabilities needs a threshold to turn them into yes/no decisions. The ROC curve shows how the model behaves at every threshold, and AUC summarises it in one number.
Two rates on the axes
- True positive rate (recall): of the actual positives, how many did we catch?
- False positive rate: of the actual negatives, how many did we wrongly flag?
The ROC curve plots true positive rate (y-axis) against false positive rate (x-axis) as the threshold moves from high to low.
Moving along the curve
- Very high threshold: almost nothing is flagged. Both rates near 0: the bottom-left corner.
- Very low threshold: almost everything is flagged. Both rates near 1: the top-right corner.
- In between: the curve shows the trade-off. A good model rises steeply, catching many positives before raising many false alarms.
A diagonal line from corner to corner is what random guessing would give.
AUC: area under the curve
AUC ranges from 0.5 (random) to 1.0 (perfect). Its plain-language meaning: the probability that the model gives a randomly chosen positive a higher score than a randomly chosen negative. It measures how well the model ranks cases, independent of any threshold.
from sklearn.metrics import roc_auc_score, roc_curve
proba = model.predict_proba(X_test)[:, 1]
auc = roc_auc_score(y_test, proba)
fpr, tpr, thresholds = roc_curve(y_test, proba)When AUC is especially useful
- Comparing models before choosing a threshold.
- Ranking problems: which leads to call first, which accounts to review first.
When to prefer the precision-recall curve
With rare positives (fraud, defects), the false positive rate can look tiny even when the absolute number of false alarms is large, so ROC curves can look flattering. The precision-recall curve and its area (average precision) focus on the positive class and are usually more informative in these cases.
Choosing the threshold
AUC does not choose a threshold for you. Decide based on business costs:
- If false alarms are cheap and misses are costly (patient follow-up calls), choose a lower threshold for higher recall.
- If each action is expensive (field visits), choose a higher threshold for higher precision.
- If capacity is fixed ("we can call 200 people a day"), pick the threshold that selects the top 200 scores and measure precision there.
Common mistakes
- Reporting AUC on training data.
- Treating AUC as accuracy.
- Ignoring calibration: a model can rank well (high AUC) while its probabilities are poorly calibrated.
Practise
Move the threshold on the metrics page and watch the confusion matrix, precision and recall change. Read the confusion matrix guide, then tune a threshold for a sales team in the trial users practice set.
Practise what you just read
Reading helps; writing the code yourself is what sticks. Start here: