Naive Bayes: A Probabilistic Classifier

Bayes' rule, run backwards

Classification asks: given a point's features, which class is most likely? Bayes' rule flips the question around into one that is often easier to answer directly - given the class, how likely are these features? - and combines it with how common each class is overall. For a point x and a class c, the posterior probability is proportional to the class's prior probability times the likelihood of x under that class: P(c given x) is proportional to P(c) times P(x given c). To classify a new point, compute this quantity for every class and pick the largest.

live demo · posterior decision boundary between two classes● LIVE

Where naive comes in

With more than one feature, P(x given c) means the joint likelihood of every feature together, which in general requires modelling how the features correlate with each other - expensive and data-hungry. The naive assumption sidesteps this entirely: it treats every feature as conditionally independent given the class, so the joint likelihood factors into a simple product of one-dimensional likelihoods, one per feature. This assumption is almost always technically false, yet naive Bayes remains a strong, fast baseline because classification only needs the largest posterior to be correct, not the exact probability - the ranking across classes tends to survive the independence shortcut even when the probabilities themselves would not.

The Gaussian version

For continuous features, Gaussian naive Bayes models each feature within each class as a normal distribution, estimated straight from the training data: the mean and variance of that feature among the points labelled with that class, nothing more. Training is therefore just arithmetic - a pass over the data to compute per-class, per-feature means and variances - with no iterative optimisation at all.

function gaussianLikelihood(x, mean, variance) {
  const exponent = -((x - mean) ** 2) / (2 * variance);
  return Math.exp(exponent) / Math.sqrt(2 * Math.PI * variance);
}
function classify(point, classes) {
  let best = null, bestScore = -Infinity;
  for (const c of classes) {
    let logScore = Math.log(c.prior);
    for (let d = 0; d < point.length; d++)
      logScore += Math.log(gaussianLikelihood(point[d], c.mean[d], c.variance[d]));
    if (logScore > bestScore) { bestScore = logScore; best = c; }
  }
  return best;
}

The shape of the decision boundary

The boundary between two classes is where their posteriors are equal. Taking logs turns the Gaussian densities into quadratic expressions in x, so in the general case, where classes have different variances, the boundary is a curve - a parabola, ellipse, or hyperbola depending on the relative variances. If two classes happen to share the same variance in every dimension, the quadratic terms cancel exactly and the boundary collapses to a straight line, which is why Gaussian naive Bayes is sometimes mistaken for a purely linear classifier - it is linear only in that special, symmetric case.

Frequently asked questions

Why is it called naive?

Because it assumes every feature is conditionally independent of every other feature given the class - a simplification that is almost never exactly true. A point's x and y coordinates are usually correlated within a real cluster, but the classifier multiplies their individual likelihoods as if they were not, and in practice this rarely hurts the final classification much.

Is the decision boundary always a straight line?

Only when every class shares the same variance in every feature. In that special case the quadratic terms in the log-posteriors cancel and the boundary is linear. When classes have different variances, which is the general case, the boundary is a curve - typically a parabola, ellipse or hyperbola.

How does naive Bayes handle a class it has never seen a feature value for?

For Gaussian features this is rarely a hard problem because a Gaussian density is never exactly zero, only very small far from the mean. For discrete features, such as word counts in text classification, a genuinely unseen value would multiply the whole product by zero, so implementations add Laplace smoothing - a small constant added to every count - so no probability is ever exactly zero.

Try it live

Everything above runs in your browser - open Naive Bayes Classifier and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.

▶ Open Naive Bayes Classifier simulation