Concept30 minBefore this:Probability Basics & Bayes' Rule

Naive Bayes

Its core assumption is almost always wrong. Why does it still classify so well?

The situation

You're building a spam filter from 100 messages. Almost every message is unique, so you can't just count how often each whole message is spam. The interviewer asks:

What is naive about naive Bayes?

Running example: a small spam filter. Its training mail has 40 spam and 60 ham messages, and a vocabulary of six words. The counts below are illustrative (made up to be easy to follow), and every probability on the page is computed live from them. Later chapters use simulated numeric data for Gaussian naive Bayes.

Reverse the question

word	free	win	offer	meeting	project	lunch	total
count in spam	30	20	25	2	3	0	80
count in ham	4	1	5	30	35	25	100

We want P(spam | this message). That's hard to estimate directly: almost every message is unique. The reverse is easy to count: how often does the word "free" appear in spam? Bayes' rule turns the easy direction into the one we need. The only obstacle is that a message has many words, and counting every combination of words is hopeless. Naive Bayes removes that obstacle with one bold assumption.

The idea

Think of a detective who treats every clue as if it were independent of the others. That's rarely true, because clues are often linked. But it makes the case easy to work through, and the detective usually names the right suspect, just with too much confidence.

From the lesson's spam example, with Laplace smoothing.

Play

Try to break it. Build a message with the word "lunch" in it, and set the smoothing as low as it goes. Then raise it.

Smoothing

One unseen word must not veto everything

"lunch" never appeared in our spam, so the raw estimate P(lunch | spam) = 0/80 = 0. Any message containing "lunch" then gets P(spam) = 0, however spammy its other words. The fix is to add α imaginary occurrences of every word:

scikit-learn: the smoothing priors "account for features not present in the learning samples and prevent zero probabilities" naive Bayes guide.

Build a message (click a word to add it; click "clear" to start again)

Smoothing α

–

P(spam | message)

–

verdict

–

Bars: each word's log-likelihood ratio, log P(word | spam) − log P(word | ham). Right = evidence for spam, left = for ham. Set α = 0 and pick "free win lunch".

With α = 0, "free win lunch" scores P(spam) = 0, a certain "ham", even though two of its three words are strongly spammy: "lunch" vetoes the spam class. With α = 0.1 it's 0.41, and with Laplace α = 1 it's 0.76. Words that are never seen in either class at prediction time are simply skipped.

Now say why: why can one unseen word overrule everything else in the message?

Maths

In one breath

Naive Bayes classifies by asking "which class makes this input most likely?", using Bayes' rule. It makes one big simplification: given the class, every feature is independent, so the likelihood is just a product of one-feature probabilities you can count from data. That makes it tiny, instant to train, and good with little data, which is why it was the classic spam filter. Two practical fixes are mandatory: smoothing (an unseen word must not zero out a class) and log space (products of many small numbers underflow). The independence assumption is almost always false, yet the ranking of classes often survives; the probabilities don't: they're pushed towards 0 and 1.What you'll be able to do after this: classify a message by hand, explain smoothing with a concrete failure, pick the right variant, and explain why naive Bayes can be accurate but badly calibrated.

spamham (normal mail)the naive assumptionnumbersfix / winfull model (no assumption)

Bayes' rule, term by term

One principle

Given the class, treat every feature as independent

scikit-learn: naive Bayes applies Bayes' theorem "with the 'naive' assumption of conditional independence between every pair of features given the value of the class variable" naive Bayes guide.

Swipe sideways to see the whole diagram.

A new message with three words from our vocabulary.
Start each class with its prior: 40% of training mail was spam, 60% ham.
Multiply by P("free" | class). With Laplace smoothing (try it in the Play lab): spam (30 + 1)/(80 + 6), ham (4 + 1)/(100 + 6).
Then "offer" and "meeting", each as if it were independent of the others given the class.
Compare the two products. Divide each by their sum to get probabilities: about 0.76 spam. "free" and "offer" pull towards spam; "meeting" pulls hard towards ham.

Work in logs

Products underflow; sums don't

A 1,000-word email multiplies 1,000 probabilities, each maybe 0.001. The product, 10⁻³⁰⁰⁰, is far below the smallest positive double (about 10⁻³⁰⁸), so the computer stores 0 for both classes and can't compare them. Taking logs turns the product into a sum, which keeps the comparison exact:

Seen this way, Multinomial naive Bayes is a linear model in the word counts: each word has a weight (its log probability per class). It's the same functional form as logistic regression; only the way the weights are found differs (counting vs optimising).

Numeric features

Gaussian naive Bayes: one bell curve per feature per class

scikit-learn's GaussianNB likelihood (σ and μ per class and feature) naive Bayes guide.

Multiplying per-feature bell curves is a Gaussian with a diagonal covariance: its contours are ellipses aligned with the axes. It can't represent "these two features move together". Raise the correlation below and compare with a full-covariance Gaussian model (quadratic discriminant analysis), which can:

Gaussian NB test accuracy

–

full-covariance Gaussian

–

Shading: Gaussian NB's decision regions. Ellipses: the 2σ contour each model fits for each class (orange dashed = NB, axis-aligned; violet = full covariance). Simulated: 200 training and 4,000 test points.

With uncorrelated features both models score 76.1%. At correlation 0.95, naive Bayes gets 95.0% and the full model 99.9%: the classes are separated along a diagonal direction that only the correlation reveals. Naive Bayes still does reasonably, which is the theme of this page: wrong assumption, decent decisions.

In practice

Generative vs discriminative

There are two ways to build a classifier. Discriminative models (logistic regression, SVMs, trees) learn P(y | x) or the boundary directly. Generative models learn how each class produces data, P(x | y) and P(y), then use Bayes' rule. Naive Bayes is the simplest generative classifier. Ng & Jordan (2001) compared it with its discriminative twin, logistic regression: logistic regression has the lower error with lots of data, but naive Bayes "may also approach its (higher) asymptotic error much faster" NIPS 2001. With little data naive Bayes often wins; with lots, logistic regression catches up and passes it.

Pick the right likelihood

The naive Bayes family

Variant	Feature type	Notes
GaussianNB	Continuous	Per-class mean and variance per feature
MultinomialNB	Counts (word counts, TF-IDF also works in practice)	The classic text classifier; smoothing α
BernoulliNB	Binary present/absent	"explicitly penalizes the non-occurrence of a feature i that is an indicator for class y, where the multinomial variant would simply ignore a non-occurring feature"
ComplementNB	Counts, imbalanced classes	Uses statistics from the complement of each class; "regularly outperforms MNB (often by a considerable margin) on text classification tasks" (Rennie et al. 2003)
CategoricalNB	Categorical (ordinal-encoded)	Its own categorical distribution per feature

All from scikit-learn's naive Bayes guide docs. Every variant supports partial_fit, so it can learn from a stream in batches.

A text classifier, step by step

Vectorise: word counts (CountVectorizer) or TF-IDF.
Choose the variant: Multinomial or Complement for counts, Bernoulli for short texts as presence/absence, Gaussian for numeric features.
Tune α by cross-validation (try 0.01–1).
Train: one pass of counting, O(n × features).
Calibrate if you'll threshold or average the probabilities.

When to use it

Use naive Bayes when

Text classification baseline: spam, topics, sentiment, routing tickets
Very little training data, or data arrives as a stream
You need training and prediction to be nearly free

Reach for something else when

You need trustworthy probabilities (calibrate or use logistic regression)
Features interact strongly: trees, boosting
Plenty of data: logistic regression, boosting or fine-tuned transformers usually win on text

The answer that sounds right

The trap

"Just count P(x | y) for whole messages"

Modelling P(x | y) exactly means estimating the probability of every possible combination of words. With only our 6 words treated as present/absent there are 2⁶ = 64 combinations per class; with a realistic 50,000-word vocabulary there are 2^50,000. You will never see most combinations even once, so their counted probability is 0, and any new message would get probability zero under both classes. Exact Bayes is right in theory and impossible in practice.

When the assumption bites

Correlated features are counted twice

Suppose a tokenizer produces "free", "FREE" and "free!" as separate features, and they always appear together. They're one piece of evidence, but naive Bayes treats them as three independent ones and multiplies the same likelihood ratio three times. The class it picks often stays the same; the confidence explodes.

Copies of the same feature

accuracy (20,000 cases)

–

log loss (lower is better)

–

share of predictions > 95% sure

–

…and how often those are right

–

Simulated: one numeric feature, class 0 ~ N(0, 1), class 1 ~ N(1.5, 1), equal priors. Violet: the true probability. Red: naive Bayes after copying the feature k times. Same 0.5 crossing, so the same decisions; a steeper, overconfident curve.

Accuracy stays at 77.4% for any number of copies, while log loss rises from 0.47 to 1.02 at 5 copies, and the predictions it is "more than 95% sure" about are right only 83.8% of the time. This is Domingos & Pazzani's point: naive Bayes can be optimal under zero–one loss even when the independence assumption is clearly violated, because classification only needs the right class to come out on top, not the right probability Machine Learning 29. scikit-learn: naive Bayes "is known to be a decent classifier, [but] it is known to be a bad estimator, so the probability outputs from predict_proba are not to be taken too seriously" naive Bayes guide.

The fix when you need probabilities

Calibrate the outputs on held-out data (CalibratedClassifierCV with isotonic or sigmoid calibration), or switch to logistic regression. Ranking metrics like ROC-AUC are unaffected by this kind of overconfidence; log loss and Brier score are not. See Probability metrics.

Caveats

Strengths

Trains in one counting pass; predicts instantly; tiny model
"require a small amount of training data" (scikit-learn)
Handles thousands of sparse features; streams with partial_fit
A strong, honest baseline for text

Weaknesses

Overconfident, poorly calibrated probabilities
Double-counts correlated features
Can't model interactions ("free" is spammy unless next to "lunch")
Usually beaten by logistic regression or boosting once data is plentiful

Say it

What is naive about naive Bayes?

Say your answer out loud

Aim for 30 seconds. It feels different from reading it, and that's the point.

Tick what you covered

It's a generative classifier built on Bayes' rule.
It assumes features are conditionally independent given the class. That's the naive part.
So the likelihood factorises into per-feature probabilities you get by counting.
You smooth the counts, so unseen features don't give zero probability.
You compute in log space, to avoid underflow.
It double-counts correlated evidence, so its probabilities are overconfident.