The Naive Bayes Classifier

I’ve needed to derive and use the naive Bayes classifier a few times over the past years, so for my own reference I’m writing down here all of the steps in the derivation of the classifier along with a couple of example use cases so that I can refresh my memory as needed.

The Problem

The problem we are solving is as follows: Imagine a training dataset described by features, . This training set is classified by hand into one of the classes . The classifier needs to, given a new set of features, predict which of the classes the new data corresponds to. That is, we want to calculate .

The Naive Bayesian Probabalistic Model

In the case where there are millions of features, or many values that each feature can take, looking up in probability tables is not feasible, so we turn to Bayes rule:

Only the numerator of the fraction is interesting for classification, the denominator is just a constant number that depends on the particular set of features chosen, so:

We can make the numerator that remains more tractable by noting that we can break down a bunch of joint probabilities through repeated application of the definition of conditional probability. Concretely, for the case of four variables, through :

More generally, for the case of variables:

We use this to write for our problem:

Now we need to make the assumption that each of the features is conditionally independent of all of the other features. That is, and and so on. This removes all of the conditions above and we can simplify the previous equation to

The Binary Classification Problem

So far we have derived the naive Bayes probability model, we need to add a decision rule to make a classifier. One very common choice is just to pick the most probable class (the maximum a posteriori or MAP decision rule). In general, we do not expect the features to be independent, but the naive Bayesian model is still of a surprising amount of use. In general we don’t expect the values for the probabilities of class membership to be accurate if the features are dependent, but as long as they’re in the correct order the MAP decision rule will still return the correct.

Imagine now a situation where, for a given set of features we wish to decide whether a particular entry belongs to a class () or does not (). We begin by defining the likelihood ratio:

Here, and represent our knowledge of the prior probabilities. This prior probability distribution might be based on our knowledge of frequencies in the larger population, or on frequency in the training set. It is also important to note that the product is carried out over all features, including those that are negative in the sample we’re considering.

If the likelihood ratio is greater than 1, the naive Bayesian classifier predicts that a given entry belongs to class . In practice, there are frequently many thousands of features and the probabilities of any given feature being present may be very small, so we work in terms of the log likelihood

Worked Example: Sentiment Analysis

Imagine that you have the following set of messages about a mobile phone


We wish to be able to automatically classify new messages that we receive either as having a positive or negative sentiment. From the overall population, and . We can now calculate the conditional probabilities for each word being in a positive and negative message.


Note here that we have employed ‘smoothing’ or ‘pseudocounting’. Where a term doesn’t appear in a given class (e.g. the word hate appears in no positive messages) we don’t want to return a zero as it sends the likelihood calculation haywire. Instead we make it the smallest possible number. Now consider trying to automatically classify the following message


The prefactors outside the fromt of the likelihood ratio cancel out (), and multiplying out the remaining products gives


I don’t know which way this works out, but it highlights one big weakness with naive Bayes. That is that we have assumed the message is a bag of words, where their relative positions are unimportant. For example, the presence of no in front of pleasure makes this a negative statement that naive Bayes interprets incorrectly.

comments powered by Disqus