Gaussian Mixture Models (GMM)

Hard vs Soft Clustering

Hard Clustering (K-Means, DBSCAN)

Each point belongs to exactly ONE cluster.

Point: [5.1, 3.5] → Cluster 0 (100%)

Soft Clustering (GMM)

Each point has PROBABILITY of belonging to each cluster.

Point: [5.1, 3.5] → 70% Cluster 0, 30% Cluster 1

The Model

Assume each cluster is a Gaussian distribution (bell curve):

• Cluster A: Mean=μ_A, Covariance=Σ_A
• Cluster B: Mean=μ_B, Covariance=Σ_B
• ...

A data point is sampled from one of these Gaussians!

Graphically:

Two overlapping bell curves.
Point near the overlap belongs to both with high probability.

EM Algorithm (Expectation-Maximization)

1. Initialization: Randomly place K Gaussians
2. E-step (Expectation): For each point, calculate probability of belonging to each Gaussian
3. M-step (Maximization): Update Gaussian parameters (μ, Σ) based on probabilities
4. Repeat until convergence

Why it works:

• E-step: "Which cluster is this point from?"
• M-step: "Refit each cluster to its assigned points"
• Iterate until stable

Advantages & Disadvantages

Pros:

• ✓ Probabilistic (know confidence)
• ✓ Can handle overlapping clusters
• ✓ More flexible than K-Means
• ✓ Theoretical foundation

Cons:

• ✗ Assumes Gaussian shape (may not hold)
• ✗ Sensitive to number of components (K)
• ✗ Slower than K-Means
• ✗ Can get stuck in local optima

Choosing Number of Clusters

Use AIC (Akaike Information Criterion) or BIC (Bayesian Information Criterion):

• Train GMM with K=1,2,3,... up to 10
• Calculate AIC/BIC for each
• Lower is better
• Pick K with lowest BIC

GMM vs K-Means vs DBSCAN