XGBoost — eXtreme Gradient Boosting

Gradient Boosting Recap

1. Train a weak learner (shallow tree)
2. Calculate residuals (errors)
3. Train next tree to predict residuals
4. Repeat N times, combining all trees

XGBoost = Optimized Gradient Boosting with:

• Parallel processing
• GPU support
• Built-in regularization
• Handles missing values
• Feature importance built-in
• Early stopping

Why Kaggle Winners Use It

Statistics:

• ~80% of Kaggle competition winners use XGBoost
• beats "pure" Deep Learning on tabular data
• Twice as fast as standard Gradient Boosting

Hyperparameters

Pro Tips:

1. Start with max_depth=5, learning_rate=0.1, n_estimators=100
2. Use early_stopping to avoid overfitting
3. Use feature_importances_ for feature selection
4. Set scale_pos_weight = n_negatives / n_positives for imbalanced data