Gradient Boosting — XGBoost, LightGBM, Tuning & Calibration

What is Gradient Boosting?

Gradient boosting builds an additive ensemble of weak learners — almost always shallow decision trees — where each new tree is fit to the negative gradient of the loss with respect to the current ensemble's predictions. In plain terms: every new tree tries to correct the mistakes of the previous ones.

Formally, at iteration t the model is F_t(x) = F_(t-1)(x) + η · h_t(x), where h_t is the new tree fit to -∂L/∂F_(t-1)(x) and η is the learning rate. For squared loss the gradient is just the residual y - F_(t-1)(x); for logistic loss it is y - σ(F_(t-1)(x)).

XGBoost

XGBoost (Extreme Gradient Boosting) popularised gradient boosting at scale. Its key innovations over classical GBM are a regularised objective, second-order (Newton) updates, and a highly engineered implementation.

Regularised objective

XGBoost optimises Obj = Σ L(y_i, ŷ_i) + Σ Ω(f_k) where Ω(f) = γ·T + ½·λ·||w||². T is the number of leaves and w are the leaf weights, so γ penalises tree complexity and λ shrinks leaf scores. This built-in regularisation is why XGBoost overfits less than classical GBM at the same depth.

Second-order split finding

Each split is scored with both the gradient g and Hessian h of the loss. The optimal leaf weight is w* = -Σg / (Σh + λ) and the gain of a split is computed in closed form. This is more accurate than first-order gradient descent and converges in fewer trees.

Key XGBoost hyperparameters

• n_estimators — number of boosting rounds. Use early stopping rather than tuning this directly.
• learning_rate (eta) — shrinkage applied to each new tree. Lower = more trees needed but better generalisation. Typical: 0.01–0.1.
• max_depth — depth of each tree. Typical: 3–8. Higher = more interactions captured but more overfitting.
• min_child_weight — minimum sum of Hessians in a leaf. Acts like a minimum-samples constraint; raise to fight overfitting.
• subsample — row sampling per tree (0.6–1.0). Adds stochasticity, reduces variance.
• colsample_bytree — feature sampling per tree (0.6–1.0).
• reg_alpha (L1) and reg_lambda (L2) — leaf-weight regularisation.
• gamma — minimum loss reduction required to make a split. Higher = more conservative trees.
• scale_pos_weight — class imbalance handling for binary classification. Set to neg / pos ratio.

XGBoost — canonical training loop

import xgboost as xgb
from sklearn.model_selection import train_test_split

X_tr, X_va, y_tr, y_va = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

model = xgb.XGBClassifier(
    n_estimators=2000,
    learning_rate=0.05,
    max_depth=6,
    min_child_weight=1,
    subsample=0.8,
    colsample_bytree=0.8,
    reg_lambda=1.0,
    objective="binary:logistic",
    eval_metric="aucpr",
    tree_method="hist",        # fast histogram algorithm
    early_stopping_rounds=50,
    random_state=42,
)

model.fit(X_tr, y_tr, eval_set=[(X_va, y_va)], verbose=False)
print("Best iteration:", model.best_iteration)
print("Best AUC-PR:", model.best_score)

LightGBM

LightGBM (Microsoft) is built on the same gradient-boosting math as XGBoost but introduces two algorithmic changes that make it 5–20× faster on large datasets with similar or better accuracy.

Leaf-wise tree growth

Where XGBoost grows trees level-wise (all nodes at depth d before depth d+1), LightGBM grows leaf-wise: it always splits the leaf with the highest loss reduction. This produces deeper, asymmetric trees that fit the data better per node — but also overfits faster on small data. Control it with num_leaves rather than max_depth.

GOSS and EFB

• GOSS (Gradient-based One-Side Sampling) — keeps all rows with large gradients (under-trained samples) and randomly samples rows with small gradients. Speeds up training without much accuracy loss.
• EFB (Exclusive Feature Bundling) — bundles mutually exclusive sparse features (those that rarely take non-zero values simultaneously) into a single feature. Huge speedup on one-hot/categorical data.

Key LightGBM hyperparameters

• num_leaves — the primary capacity knob. Rule of thumb: num_leaves ≤ 2^max_depth. Typical: 31–255.
• learning_rate — same role as XGBoost; 0.01–0.1.
• min_data_in_leaf — critical anti-overfitting parameter. Typical: 20–500. Raise on small/noisy datasets.
• feature_fraction and bagging_fraction — column / row sampling.
• lambda_l1, lambda_l2 — leaf regularisation.
• max_bin — number of histogram bins for continuous features. Lower = faster, slightly less accurate.
• categorical_feature — pass categorical columns natively; LightGBM uses Fisher's optimal partitioning instead of one-hot.

LightGBM — canonical training loop

import lightgbm as lgb

train_set = lgb.Dataset(X_tr, label=y_tr, categorical_feature=["city", "device"])
val_set   = lgb.Dataset(X_va, label=y_va, reference=train_set)

params = {
    "objective": "binary",
    "metric": "average_precision",
    "learning_rate": 0.05,
    "num_leaves": 63,
    "min_data_in_leaf": 100,
    "feature_fraction": 0.8,
    "bagging_fraction": 0.8,
    "bagging_freq": 5,
    "lambda_l2": 1.0,
    "verbose": -1,
}

model = lgb.train(
    params,
    train_set,
    num_boost_round=5000,
    valid_sets=[val_set],
    callbacks=[lgb.early_stopping(50), lgb.log_evaluation(100)],
)
print("Best iter:", model.best_iteration)

XGBoost vs LightGBM — when to use which

• LightGBM — large datasets (> 100k rows), many categorical features, latency-sensitive training. Default choice for most modern tabular problems.
• XGBoost — small/medium data, when you want maximum stability and the most mature ecosystem, or when leaf-wise growth overfits your problem.
• CatBoost — heavy categorical data with target leakage concerns; uses ordered boosting and is often best out-of-the-box without tuning.

Hyperparameter Tuning

There is a sane order to tune gradient-boosting models. Random or grid search across all parameters at once wastes compute; do it in stages.

Recommended tuning order

• 1. Fix a low learning rate (0.05) and use early stopping to find a baseline.
• 2. Tune tree complexity — max_depth / num_leaves, min_child_weight / min_data_in_leaf.
• 3. Tune stochasticity — subsample, colsample_bytree / feature_fraction.
• 4. Tune regularisation — reg_alpha, reg_lambda, gamma.
• 5. Lower the learning rate (e.g. to 0.01) and increase early-stopping rounds to squeeze final accuracy.

Bayesian optimisation with Optuna

import optuna, lightgbm as lgb
from sklearn.model_selection import StratifiedKFold
import numpy as np

def objective(trial):
    params = {
        "objective": "binary",
        "metric": "average_precision",
        "learning_rate": trial.suggest_float("lr", 0.01, 0.1, log=True),
        "num_leaves": trial.suggest_int("num_leaves", 15, 255),
        "min_data_in_leaf": trial.suggest_int("min_data", 20, 500),
        "feature_fraction": trial.suggest_float("ff", 0.5, 1.0),
        "bagging_fraction": trial.suggest_float("bf", 0.5, 1.0),
        "bagging_freq": 5,
        "lambda_l1": trial.suggest_float("l1", 1e-3, 10.0, log=True),
        "lambda_l2": trial.suggest_float("l2", 1e-3, 10.0, log=True),
        "verbose": -1,
    }
    cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
    scores = []
    for tr_idx, va_idx in cv.split(X, y):
        dtr = lgb.Dataset(X.iloc[tr_idx], y.iloc[tr_idx])
        dva = lgb.Dataset(X.iloc[va_idx], y.iloc[va_idx])
        m = lgb.train(params, dtr, num_boost_round=2000, valid_sets=[dva],
                      callbacks=[lgb.early_stopping(50, verbose=False)])
        scores.append(m.best_score["valid_0"]["average_precision"])
    return np.mean(scores)

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=100, timeout=3600)
print(study.best_params)

Probability Calibration

Tree-based models produce scores, not true probabilities. XGBoost and LightGBM outputs from predict_proba are usually pushed toward 0 and 1 (over-confident), especially after many boosting rounds. If you use the score for thresholding only, calibration may not matter. If you make business decisions on the probability itself — expected cost, expected revenue, risk scoring — you must calibrate.

How to diagnose miscalibration

• Reliability diagram — bin predictions by predicted probability and plot mean predicted vs. observed frequency. The diagonal is perfect.
• Brier score — mean squared error between predicted probability and the binary outcome. Lower is better; decomposes into calibration + refinement.
• Expected Calibration Error (ECE) — weighted average of |confidence − accuracy| over bins.

from sklearn.calibration import calibration_curve
import matplotlib.pyplot as plt

prob_true, prob_pred = calibration_curve(y_val, y_prob, n_bins=10, strategy="quantile")
plt.plot(prob_pred, prob_true, marker="o", label="Model")
plt.plot([0, 1], [0, 1], "k--", label="Perfect")
plt.xlabel("Mean predicted probability"); plt.ylabel("Observed frequency"); plt.legend()

Calibration methods

• Platt scaling (sigmoid) — fits a logistic regression on the model's scores. Works well when miscalibration is sigmoid-shaped and the dataset is small.
• Isotonic regression — fits a non-parametric monotonic mapping. More flexible than Platt; needs more data (≥ 1000 calibration samples) to avoid overfitting.
• Temperature scaling — single scalar T applied to logits before the sigmoid/softmax. Common for neural nets, occasionally useful for boosting.
• Beta calibration — parametric alternative to isotonic that often beats both Platt and isotonic on imbalanced data.

Calibrate properly with a held-out set

from sklearn.calibration import CalibratedClassifierCV
from sklearn.model_selection import train_test_split
import lightgbm as lgb

# Split into train / calibration / test — calibrator MUST see fresh data
X_tr, X_tmp, y_tr, y_tmp = train_test_split(X, y, test_size=0.4, stratify=y, random_state=0)
X_cal, X_te, y_cal, y_te = train_test_split(X_tmp, y_tmp, test_size=0.5, stratify=y_tmp, random_state=0)

base = lgb.LGBMClassifier(n_estimators=500, learning_rate=0.05, num_leaves=63)
base.fit(X_tr, y_tr)

# 'prefit' tells sklearn not to refit the base estimator
calibrated = CalibratedClassifierCV(base, method="isotonic", cv="prefit")
calibrated.fit(X_cal, y_cal)

from sklearn.metrics import brier_score_loss
print("Raw   Brier:", brier_score_loss(y_te, base.predict_proba(X_te)[:, 1]))
print("Calib Brier:", brier_score_loss(y_te, calibrated.predict_proba(X_te)[:, 1]))

Calibration and class imbalance

If you resampled (SMOTE, undersampling) or used scale_pos_weight during training, the model's output probabilities are biased toward the resampled distribution. You must either calibrate on the original distribution, or apply a prior-correction formula before calibration:

# Prior correction: undo the effect of resampling on the score
# p_true = p_resampled * (p_orig / p_resampled) /
#          (p_resampled * (p_orig / p_resampled) + (1 - p_resampled) * ((1 - p_orig) / (1 - p_resampled)))
import numpy as np

def correct_prior(p, prior_train, prior_real):
    a = p * (prior_real / prior_train)
    b = (1 - p) * ((1 - prior_real) / (1 - prior_train))
    return a / (a + b)

Production Checklist

• ✅ Early stopping on a held-out validation set, not a fixed n_estimators.
• ✅ CV scheme matches the deployment scenario (stratified / grouped / time-based).
• ✅ Tuned with Optuna or Hyperopt — not by hand, not full grid search.
• ✅ Calibrated on a separate split if probabilities drive decisions.
• ✅ Feature importance sanity-checked with SHAP, not just gain.
• ✅ Model size and inference latency measured (LightGBM num_leaves drives both).
• ✅ Monitoring in place for distribution drift in inputs and predicted-probability drift in outputs.