Gaussian Mixture Models (GMM)

1. Intuition: Why GMM Instead of K-Means?

K-Means assumes clusters are spherical, equally sized, and assigns each point to exactly one cluster. Real data rarely behaves like that. A Gaussian Mixture Modelassumes the data was generated by sampling from K different Gaussian distributions and gives each point a probability of belonging to each component — a soft assignment.

• K-Means: hard label, spherical clusters, equal variance.
• GMM: soft probabilities, elliptical clusters of arbitrary orientation, different sizes.
• K-Means is actually a special case of GMM (identity covariance + hard E-step).

2. The Generative Model

GMM assumes each point x was produced by:

• Pick a component k with probability πₖ (the mixing weight).
• Sample x ~ 𝒩(μₖ, Σₖ) from that component's Gaussian.

The resulting density is a weighted sum of Gaussians:

Parameters to learn: mixing weights πₖ, means μₖ, and covariance matrices Σₖ for each of the K components.

3. The Multivariate Gaussian Density

The covariance matrix Σ controls the cluster's shape: diagonal elements give variance along each axis, off-diagonal elements give correlations (rotation of the ellipse).

4. The EM Algorithm — Fitting a GMM

Maximum-likelihood estimation of GMM parameters has no closed form, so we use the Expectation–Maximization (EM) algorithm. EM alternates between estimating soft cluster memberships (E) and updating the Gaussian parameters (M), and is guaranteed to increase the data log-likelihood at every iteration.

E-step — Compute Responsibilities

For every point xᵢ and component k, compute the posterior:

γᵢₖ is the "responsibility" component ktakes for point i — a number in [0, 1], summing to 1 over k.

M-step — Update Parameters

Treat the responsibilities as soft counts and recompute weighted means and covariances:

Repeat Until Convergence

Iterate E and M steps until the log-likelihood stops increasing meaningfully, or for a maximum number of iterations.

5. Covariance Types

Choosing the covariance structure trades flexibility against the number of parameters and risk of overfitting:

• full — each component has its own arbitrary covariance matrix. Most flexible, most parameters.
• tied — all components share one covariance matrix. Different means, same shape.
• diag — diagonal covariance per component (axis-aligned ellipses).
• spherical — single variance per component (circles). Closest to K-Means.

6. Choosing K — Use BIC or AIC

Unlike K-Means, GMM has a proper likelihood, so we can use information criteria to choose K (and the covariance type at the same time):

Where p is the number of free parameters and Nthe number of samples. Lower is better. BIC penalizes complexity more heavily and is the usual choice for model selection in clustering.

7. Python Implementation

from sklearn.mixture import GaussianMixture
from sklearn.datasets import make_blobs
import numpy as np

X, _ = make_blobs(n_samples=600, centers=3, cluster_std=[1.0, 2.5, 0.5], random_state=42)

gmm = GaussianMixture(
    n_components=3,
    covariance_type="full",
    n_init=10,
    random_state=42,
).fit(X)

labels = gmm.predict(X)              # hard labels
proba  = gmm.predict_proba(X)        # soft responsibilities, shape (N, K)
print("weights:", gmm.weights_)
print("means:",   gmm.means_)
print("log-lik:", gmm.score(X) * len(X))

# New samples can be generated from the fitted model
X_new, y_new = gmm.sample(100)

Model Selection with BIC

bics = []
for k in range(1, 11):
    g = GaussianMixture(n_components=k, covariance_type="full",
                        n_init=5, random_state=0).fit(X)
    bics.append((k, g.bic(X)))

best_k = min(bics, key=lambda kv: kv[1])[0]
print("Best K by BIC:", best_k)

8. K-Means vs GMM at a Glance

• Cluster shape: K-Means → spheres; GMM → ellipses (any orientation).
• Assignment: K-Means → hard; GMM → soft probabilities.
• Objective: K-Means → minimize WCSS; GMM → maximize log-likelihood.
• Picks K via: elbow / silhouette vs BIC / AIC.
• Generative? K-Means → no; GMM → yes (you can sample new data).

9. Strengths & Weaknesses

Strengths

• Soft, probabilistic memberships — useful for downstream uncertainty.
• Handles elliptical, differently-sized, and overlapping clusters.
• Proper likelihood enables principled model selection (BIC/AIC).
• Generative — can sample new data and serve as a density estimator / anomaly detector.

Weaknesses

• EM is sensitive to initialization — use multiple restarts (K-Means init helps).
• Covariance matrices can become singular (a component collapses on a single point); use a regularization term (reg_covar).
• Assumes Gaussian components — fails on heavy-tailed or non-convex shapes (use DBSCAN, spectral, or Bayesian non-parametric mixtures).
• Computationally heavier than K-Means, especially with covariance_type="full" in high dimensions.

10. Practical Tips

• Standardize features before fitting — Gaussian densities are scale-sensitive.
• Initialize with K-Means++ (the scikit-learn default) for faster, more reliable convergence.
• Set n_init ≥ 5–10 to escape poor local optima.
• Use BIC across K and covariance types together to choose the model.
• For anomaly detection, threshold gmm.score_samples(X) — low log-density = outlier.
• For very large K or streaming data, consider Bayesian Gaussian Mixture (BayesianGaussianMixture) which can prune unused components automatically.