Random Forest

Intuition: Wisdom of the Crowd

A single decision tree is like asking one expert — it can be biased, overfit to quirks in the data, and unstable (small changes flip the splits). Random Forest asks hundreds of diverse experts and aggregates their answers. As long as the trees make different mistakes, averaging cancels noise and the ensemble lands closer to the truth than any single tree.

The Two Sources of Randomness

1. Bagging (Bootstrap Aggregating)

For each tree, sample n rows with replacement from the training set of size n. Each bootstrap sample contains roughly 63.2% unique rows; the rest are duplicates. The remaining ~36.8% unseen rows are called out-of-bag (OOB) samples and give a free validation set.

2. Feature Subsampling

At every split, instead of evaluating all p features, the tree considers only a random subset of size max_features. Typical defaults:

• Classification: √p
• Regression: p/3

This prevents a few dominant features from being chosen by every tree, which would make the trees correlated and defeat the purpose of averaging.

Aggregation Rules

• Classification — majority vote, or average of class probabilities (soft voting, usually better).
• Regression — arithmetic mean of tree predictions.

Why Averaging Reduces Variance

For N identically distributed predictions with variance σ² and pairwise correlation ρ, the variance of the average is:

As N → ∞, the second term vanishes — but the first term, the irreducible correlation floor, remains. That's exactly why feature subsampling matters: lower ρ means lower ensemble variance.

Out-of-Bag (OOB) Score

Each row is OOB for ~37% of the trees. Predict it using only those trees and you get an honest cross-validation-like estimate of generalization error — without holding out a separate validation set.

rf = RandomForestClassifier(n_estimators=300, oob_score=True, random_state=42)
rf.fit(X, y)
print("OOB accuracy:", rf.oob_score_)

Feature Importance

Random Forest offers two ways to rank feature relevance:

• Mean Decrease in Impurity (MDI) — sum of impurity reductions a feature provides across all splits, weighted by samples reaching the node. Fast but biased toward high-cardinality features.
• Permutation Importance — shuffle a feature's values and measure the drop in OOB or validation score. Slower, but unbiased and model-agnostic.

from sklearn.inspection import permutation_importance

result = permutation_importance(rf, X_val, y_val, n_repeats=10, random_state=42)
for i in result.importances_mean.argsort()[::-1][:10]:
    print(f"{feature_names[i]:30s} {result.importances_mean[i]:.4f}")

Python Implementation

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, GridSearchCV

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

param_grid = {
    "n_estimators": [200, 500],
    "max_depth": [None, 8, 16],
    "max_features": ["sqrt", "log2"],
    "min_samples_leaf": [1, 2, 4],
}
grid = GridSearchCV(
    RandomForestClassifier(random_state=42, oob_score=True, n_jobs=-1),
    param_grid, cv=5, n_jobs=-1, scoring="roc_auc"
)
grid.fit(X_tr, y_tr)
print("Best params:", grid.best_params_)
print("Test ROC-AUC:", grid.score(X_te, y_te))

Key Hyperparameters

• n_estimators — number of trees. More is better with diminishing returns; 100–500 is typical. Cost is linear in trees.
• max_depth — depth of each tree. None grows until pure (high variance, low bias). Cap to regularize.
• max_features — features per split. Lower → more decorrelated trees, more bias. Try sqrt, log2, or a fraction.
• min_samples_split / min_samples_leaf — minimum samples to split a node or sit in a leaf. Higher values prevent overfitting.
• bootstrap — set False for "Extra Trees" style sampling without replacement.
• class_weight — use "balanced" or "balanced_subsample" for imbalanced classes.
• n_jobs — set to -1 to parallelize across CPU cores. Trees train independently, so it scales nearly linearly.

Random Forest vs Other Tree Ensembles

• vs Single Decision Tree — much lower variance, slightly higher bias, far better generalization.
• vs Extra Trees — Extra Trees use the full dataset (no bootstrap) and pick split thresholds randomly. Faster, sometimes higher variance reduction.
• vs Gradient Boosting (XGBoost / LightGBM / CatBoost) — boosting builds trees sequentially, each correcting the previous. Usually wins on tabular benchmarks but needs more tuning and is harder to parallelize. Random Forest trains in parallel and rarely overfits with more trees.

Pros and Cons

Pros

• Strong out-of-the-box accuracy with little tuning.
• Handles mixed numeric / categorical features and missing values gracefully.
• No feature scaling required — splits are scale-invariant.
• Built-in OOB validation and feature importance.
• Trivially parallelizable; resistant to overfitting as N grows.

Cons

• Memory- and prediction-time-heavy: hundreds of deep trees can be huge.
• Less interpretable than a single tree (though SHAP helps).
• Struggles to extrapolate beyond the training range in regression.
• Often outperformed by gradient boosting on highly tuned tabular tasks.
• Default MDI importance is biased toward high-cardinality and continuous features.

When to Use Random Forest

• You need a strong baseline on tabular data with minimal tuning.
• Mixed feature types, missing values, or non-linear interactions.
• You want feature importance or OOB error for free.
• Latency / memory budget allows hundreds of trees at inference time.