Ensemble Methods — Combine Multiple Models

Why Ensembles?

"Wisdom of crowds" — multiple imperfect models often beat a single perfect one. This is why:

• Kaggle competitions: ~100% of winners use ensembles
• Real production systems: Google, Netflix, Amazon all use ensembles

Three Main Approaches

1. Bagging (Bootstrap Aggregating)

   • Train independent models on random subsets of data
   • Average predictions
   • Example: Random Forest

2. Boosting

   • Train models sequentially, each correcting previous errors
   • Weight incorrect predictions higher
   • Example: XGBoost, Gradient Boosting

3. Stacking

   • Train multiple different models
   • Use their outputs as input to a meta-learner

The Bias-Variance Tradeoff

Random Forest — The Gold Standard

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(n_estimators=100, max_depth=10)
rf.fit(X_train, y_train)
accuracy = rf.score(X_test, y_test)

Why Random Forest wins:

• Parallelizable (train trees independently)
• Handles missing data well
• Feature importance built-in
• Minimal hyperparameter tuning