Concept30 minBefore this:Logistic Regression

Classification Metrics

Your fraud model is 99.8% accurate. Why is your manager unhappy?

The situation

Your fraud model is 99.8% accurate, and your manager is unhappy. About 200 of every 100,000 transactions are fraud. The interviewer asks:

Your fraud model is 99.8% accurate. Is it good?

Scenarios used throughout: card-fraud detection (0.2% of transactions are fraud), cancer screening (1% have the disease), spam filtering (40% of mail is spam), and a customer-support intent classifier with a rare but costly "cancel subscription" class. Scores come from a simple model of a classifier (illustrative); every number on the page is computed live from it.

"Our fraud model is 99.8% accurate." Your manager is unhappy. Why?

Out of every 100,000 card transactions, about 200 are fraud. A model that never flags anything is right on the 99,800 legitimate ones: 99.8% accuracy, and it catches zero fraud. One number told you nothing about the thing you care about. Classification metrics exist to ask the right question about the right mistakes.

The idea

Think of a smoke alarm that never rings. It's right almost every day, and it's useless. Precision asks how many alarms were real fires. Recall asks how many fires set off the alarm.

Every metric in this lesson is built from these four counts.

Play

Try to break it. On Fraud, slide the threshold down until recall is high, and watch precision. Then switch to Spam and do the same.

Every metric moves with the threshold

Slide the threshold, watch the trade-off

Flag if probability ≥

–

Lower the threshold and you flag more: recall rises, precision falls. Raise it and the reverse. There's no free lunch, so the right threshold depends on what each kind of mistake costs. That's the subject of Thresholds & Misclassification Costs.

Now say why: why does the same threshold move give such different precision on rare fraud and common spam?

Maths

In one breath

Every classification metric is a ratio built from four counts: true positives, false positives, false negatives and true negatives. Accuracy hides rare classes. Precision asks "when I flag, am I right?" Recall asks "of what's out there, how much do I catch?" Specificity and NPV ask the same about negatives. F1 balances precision and recall and ignores true negatives. Precision depends on how common the positive class is; recall doesn't. For many classes, macro, micro and weighted averaging answer different questions.What you'll be able to do after this: compute and explain every metric from a confusion matrix, pick the one a scenario needs, and spot the accuracy trap instantly.

true positivetrue negativefalse positive (false alarm)false negative (miss)

Four counts: the confusion matrix

Swipe sideways to see the whole diagram.

Every prediction lands in one of four boxes: what the model said (columns) versus what was true (rows).
True positive (TP): it was fraud and we flagged it.
False negative (FN): it was fraud and we missed it. Statisticians call this a type II error.
False positive (FP): it was legit and we flagged it, a false alarm. Type I error.
True negative (TN): legit and let through.
Rows add up to what's really there; columns add up to what the model flagged. Every metric on this page is some ratio of these four numbers.

One principle

Each metric divides by a different "out of what?"

Standard definitions, as in scikit-learn's metrics guide docs. Recall, sensitivity, hit rate and true positive rate are four names for one thing.

Precision depends on how rare positives are

A 95%-specific test, and most of its positives are false

Recall and specificity describe the test itself. Precision mixes the test with the base rate: when positives are rare, even a small false-positive rate on the huge negative group swamps the true positives. That's Bayes' rule:

Sensitivity (recall) Specificity Prevalence

precision (PPV)

–

per 10,000 screened: true / false positives

–

NPV

–

Screening scenario: a good test (90% sensitive, 95% specific) at 1% prevalence flags about 495 healthy people for every 90 sick ones, so precision is about 15%. That's why screening positives get a second, more specific test. Same test in a high-risk clinic (20% prevalence): precision about 82%.

One number for precision and recall

F1: the harmonic mean, and what it ignores

Precision Recall

arithmetic mean

–

F1 (harmonic mean)

–

The harmonic mean punishes imbalance; you can't buy a good F1 with one great number. If the two errors aren't equally costly, use F-beta (next lessons), not F1.

F1's blind spots

It ignores true negatives, and it changes if you swap which class you call "positive"; MCC doesn't have either problem Chicco & Jurman 2020. By default scikit-learn returns F1 = 0 when there are no positive predictions and no positives; the zero_division parameter controls this docs.

Many classes: how you average matters

Macro, micro and weighted

Support tickets go to billing (600), technical (350) or cancel subscription (50). Cancel is rare but the most expensive to miss: an unhandled cancel is a lost customer. Compare two models:

Micro-averaging pools all decisions, so with every class included it equals accuracy docs; it's dominated by big classes. Macro-averaging gives every class equal weight, so the rare class counts as much as billing. Weighted averages by support. Model A looks fine on micro and weighted, and macro exposes it.

Average	Question it answers	Use when
micro	How good is a random single decision?	Every prediction matters equally; multi-label tagging
macro	How good is the model on a typical class?	Rare classes matter as much as common ones (support intents, rare diseases)
weighted	Macro, but proportional to class size	Reporting overall quality on imbalanced data without letting tiny classes dominate
per class (None)	Everything, unsummarised	Always look at this before trusting any average

scikit-learn notes that macro "may over-emphasize the typically low performance on an infrequent class", and that a weighted F can fall outside the range between precision and recall docs.

In practice

A report, not a number

In practice you print a per-class report (scikit-learn's classification_report: precision, recall, F1 and support for every class) next to the confusion matrix, and pick one headline metric that matches the cost of mistakes in your scenario, plus a guardrail (e.g. "maximise recall subject to precision ≥ 80%"). The sections below build each piece.

Top-k accuracy, and multi-label metrics

Top-k accuracy

Correct if the true label is among the k highest-scored classes; accuracy is the k = 1 case docs. Use it when a human or a later step picks from a short list: ImageNet top-5, product category suggestions, autocomplete.

Multi-label (several tags per item)

Subset accuracy: all tags exactly right (harsh). Hamming loss: fraction of individual tag decisions that are wrong (lenient). Jaccard: |predicted ∩ true| / |predicted ∪ true| per item. Plus per-tag precision and recall with micro or macro averaging.

Worked example: one article, true tags {politics, economy, india}

Predicted tags	Subset accuracy	Jaccard	Hamming loss (5 possible tags)
{politics, economy, india}	1	1.00	0.0
{politics, economy}	0	0.67	0.2
{politics, sports}	0	0.25	0.6

Missing one of three tags already scores 0 on subset accuracy; Jaccard and Hamming give partial credit.

Which metric, in which situation

Scenario	Headline metric	Why
Cancer screening (first test)	Recall / sensitivity	A miss can be fatal; false alarms get a cheap follow-up test
Confirmatory diagnosis	Specificity, precision	A false positive means unnecessary treatment
Spam filter	Precision (at good recall)	Losing a real email is worse than letting one spam through
Fraud (0.2%)	Recall at a fixed precision (or alerts per day)	Review capacity is limited; accuracy is meaningless here
Content moderation	Precision for auto-removal, recall for the human-review queue	Two thresholds, two jobs
Support intents with a rare costly class	Macro-F1 + per-class recall	Micro and accuracy hide the rare class
Balanced classes, equal costs (e.g. cats vs dogs)	Accuracy is fine	Its weakness only appears with imbalance or unequal costs
Imbalanced, need one fair number	MCC or balanced accuracy	Use all four cells (see Thresholds & Costs)

The answer that sounds right

The trap

Accuracy: (TP + TN) / everything

accuracy of "always say negative"

–

accuracy of a decent model

–

positives the dumb model catches

With rare positives, the do-nothing baseline already scores near 100%, and a genuinely useful model can even score lower because its false alarms cost accuracy points. With balanced classes (spam), accuracy is informative again.

What breaks

Accuracy "cannot be considered a reliable measure" on unbalanced data, because it gives "an overly optimistic estimate" Chicco & Jurman 2020. Always compare against the majority-class baseline, and use metrics that look at the positive class directly.

What to keep

All of these depend on the threshold. For threshold-free comparison, use ROC-AUC or PR-AUC (next lesson).
Precision and NPV depend on prevalence, so they don't transfer between populations (a clinic vs the general public, a test set rebalanced to 50/50 vs production). Recall and specificity do transfer, if the positives themselves look the same.
Small classes give noisy metrics. Recall on 50 "cancel" tickets moves 2 points per ticket; report confidence intervals (Choosing the Right Metric).
Metrics are proxies for costs. When you can put a price on each error, optimise expected cost directly. my take

Say it

Your fraud model is 99.8% accurate. Is it good?

Say your answer out loud

Aim for 30 seconds. It feels different from reading it, and that's the point.

Tick what you covered

You start from the confusion matrix.
Accuracy misleads on imbalanced data, because the majority baseline already scores high.
Precision is how many flagged items are right, and recall is how many real positives you catch.
Which you prioritise depends on the costs: recall for screening, precision for spam.
F1 balances them but ignores true negatives.
Precision shifts with prevalence, and macro averaging helps when rare classes matter.