Naive Bayes Classification

Bayes' Theorem

P(Class | Features) = P(Features | Class) * P(Class) / P(Features)

English: Probability of class given features = likelihood × prior / evidence

Components:

• Prior P(Class): How common is this class? (Before seeing features)
• Likelihood P(Features | Class): If this class is true, how likely are these features?
• Evidence P(Features): How likely is this feature combination overall?

Example: Email Spam Detection

Email has word "FREE" → Is it spam?

• Prior: 20% of emails are spam
• Likelihood: 50% of spam emails contain "FREE"
• Likelihood (normal): 5% of normal emails contain "FREE"
• Result: With "FREE", probability of spam increases significantly

The "Naive" Assumption

Naive Bayes assumes all features are independent given the class.

P(word1, word2, word3 | spam) = P(word1|spam) × P(word2|spam) × P(word3|spam)

This is often wrong (words are correlated), but it works well in practice!

Why?

• Simplification makes computation fast
• Despite false assumption, predictions are often accurate
• Great for high-dimensional data (text)

Advantages & Disadvantages

Pros:

• ✓ Fast to train (just counting)
• ✓ Fast to predict
• ✓ Works well with high-dimensional data (text, images)
• ✓ Requires little training data
• ✓ Interpretable (see which features matter)

Cons:

• ✗ Independence assumption often violated
• ✗ Poor probability estimates (miscalibrated)
• ✗ Can struggle with rare features