You train a model to flag fraudulent card transactions. It scores 99% accuracy. Great? Not necessarily. If only a tiny share of transactions are fraud, a model that says "not fraud" for every single transaction would also score very high accuracy, while catching zero fraud. This is why analysts look beyond accuracy.
The confusion matrix
For a yes/no prediction, every case lands in one of four boxes:
| Predicted yes | Predicted no | |
|---|---|---|
| Actually yes | True positive (TP) | False negative (FN) |
| Actually no | False positive (FP) | True negative (TN) |
- True positive: fraud, and the model flagged it.
- False negative: fraud, but the model missed it.
- False positive: a genuine transaction the model wrongly flagged.
- True negative: genuine, and correctly left alone.
from sklearn.metrics import confusion_matrix, classification_report
confusion_matrix(y_test, y_pred)
print(classification_report(y_test, y_pred))Accuracy
(TP + TN) / everything. Useful only when the classes are roughly balanced and both kinds of mistake cost about the same.
Precision: when the model says yes, how often is it right?
TP / (TP + FP). High precision means few false alarms. It matters when acting on a "yes" is expensive or annoying: blocking a customer's card, sending a field agent, calling a customer.
Recall: of all the real yeses, how many did we catch?
TP / (TP + FN). High recall means few misses. It matters when missing a case is costly: a patient who will be readmitted, a fraudulent transaction, a machine about to fail.
The trade-off
Most classifiers output a probability, and you choose the threshold. Lower the threshold and you flag more cases: recall rises, precision usually falls. Raise it and the opposite happens. There is no free lunch; the right balance depends on the cost of each mistake.
Ask the business: "Which is worse, a false alarm or a miss, and by how much?" The answer tells you which metric to prioritise and where to set the threshold.
Examples
- Hospital readmission alerts: a missed patient can be serious, and a follow-up call is cheap. Favour recall.
- Blocking card transactions: each false block annoys a genuine customer. Balance both, often favouring precision at the strictest action and using softer checks (an OTP) for borderline cases.
- Sales call list with limited callers: you can only call so many people, so the top of the list must be good. Favour precision among the top-ranked.
F1 score
The harmonic mean of precision and recall: 2 × P × R / (P + R). It is high only when both are reasonably high, which makes it a decent single number for imbalanced problems when you have no clear cost information.
ROC-AUC and PR-AUC
Instead of judging one threshold, these summarise performance across all thresholds.
- ROC-AUC: the chance the model ranks a random positive above a random negative. 0.5 is random guessing; 1.0 is perfect.
- PR-AUC (precision-recall curve area) is often more informative when positives are rare.
Evaluate on unseen data
All these metrics must be calculated on data the model did not train on (a test set, or cross-validation). A model can look perfect on its training data simply by memorising it. Comparing training and test scores is the quickest check for overfitting.
Practise
Our ML metrics page lets you move a threshold and watch the confusion matrix and every metric change. Then apply it in the hospital readmissions and online returns practice sets, where choosing the right metric is part of the task.
Practise what you just read
Reading helps; writing the code yourself is what sticks. Start here: