Supervised Learning, Leakage & Validation

What is Supervised Learning?

Supervised learning is the most common paradigm in machine learning. You are given a dataset of (X, y) pairs — inputs and their known correct outputs — and your goal is to learn a function f that maps new, unseen inputs to accurate predictions. The "supervision" comes from the labels: the model is explicitly told what the right answer is during training.

Regression vs. Classification

• Regression — the target y is continuous. Examples: predicting house prices, forecasting temperature, estimating time-to-failure.
• Classification — the target y is discrete (a category). Examples: spam detection, disease diagnosis, image recognition.

Many algorithms work for both (e.g. neural networks, decision trees, k-NN). Some are specialized: linear regression is natively for regression; logistic regression and Naive Bayes are natively for classification.

The Generalization Problem

The goal is not to memorize the training data — it is to perform well on new data drawn from the same distribution. The gap between training performance and test performance is called generalization error. Your job as a practitioner is to minimize this gap, not just the training error.

The Train / Validation / Test Split

To estimate generalization, we partition data into three disjoint sets:

• Training set — used to fit model parameters (weights, splits, etc.)
• Validation set — used to tune hyperparameters and make model-selection decisions
• Test set — used once at the very end to report unbiased performance

A common split ratio is 70% / 15% / 15% or 80% / 10% / 10%, but the right ratio depends on dataset size. With millions of rows, even 1% test data can be statistically reliable.

Why three sets, not two?

If you repeatedly evaluate on the test set while tuning hyperparameters, you are effectively "training on the test set." The test set leaks information back into model selection, and your final reported number will be optimistically biased. The validation set acts as a proxy for the test set during development.

from sklearn.model_selection import train_test_split

# One-shot split: 80% train, 20% test
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, random_state=42, stratify=y  # stratify preserves class ratios
)

# Further split train into train + validation
X_train, X_val, y_train, y_val = train_test_split(
    X_train, y_train, test_size=0.15, random_state=42, stratify=y_train
)

K-Fold Cross-Validation

When data is scarce, a single train/validation split is noisy — you might get lucky or unlucky with which rows land in the validation fold. K-fold cross-validationmitigates this by rotating the validation set across K partitions.

How it works

• Split the data into K equal folds (typically K = 5 or 10)
• For each fold i, train on the other K−1 folds and validate on fold i
• Aggregate the K validation scores (usually by averaging)

The final score is the mean across folds; the standard deviation across folds tells you how stable the metric is. A high standard deviation means the model is sensitive to the specific training sample.

from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(n_estimators=100, random_state=42)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

scores = cross_val_score(model, X, y, cv=cv, scoring='roc_auc')
print(f"AUC-ROC: {scores.mean():.4f} (+/- {scores.std():.4f})")

Stratified K-Fold

In classification, especially with imbalanced classes, a random fold might contain no positive examples. Stratified K-Fold ensures each fold preserves the overall class distribution. Always use stratification for classification.

Leave-One-Out Cross-Validation (LOOCV)

A special case where K = N (one sample per fold). It is deterministic and uses almost all data for training each round, but it is computationally expensive and has high variance because the validation sets are highly correlated (they differ by only one sample). Use only for very small datasets where K-fold is unreliable.

When NOT to use K-fold

• Time-series data — shuffling breaks temporal order; use walk-forward validation instead
• Grouped data — if multiple rows belong to the same patient/customer, they must stay together; use GroupKFold
• Very large datasets — a single holdout is statistically reliable and much faster

Time-Series Validation

In time-series, the future is not available at prediction time, so random shuffling is invalid. The validation set must always come after the training set in time.

Walk-forward validation

Train on [t₀, t₁], validate on [t₁, t₂]. Then train on [t₀, t₂], validate on [t₂, t₃], and so on. This mimics how the model will actually be deployed.

from sklearn.model_selection import TimeSeriesSplit

tscv = TimeSeriesSplit(n_splits=5)
for fold, (train_idx, val_idx) in enumerate(tscv.split(X)):
    print(f"Fold {fold}: train={len(train_idx)}, val={len(val_idx)}")

Data Leakage: The Silent Killer

Data leakage occurs when information from outside the training set is used to create the model, giving an unrealistic estimate of performance. Leakage often produces suspiciously good validation scores that crash in production.

Types of leakage

1. Train-test leakage

The test set contaminates the training process. Classic examples: scaling the entire dataset before splitting; imputing missing values using global statistics that include the test set; or deduplicating after splitting so identical rows appear in both sets.

# WRONG — leakage! scaler sees test data during fit
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)           # NO!
X_train, X_test = train_test_split(X_scaled) # split AFTER scaling

# RIGHT — fit scaler only on training data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)      # only transform!

2. Target leakage

A feature is derived from information that would not be available at prediction time. Example: predicting whether a patient has a disease, but one feature is "medication prescribed" — the prescription happens after diagnosis, so the model is learning to recognize treatment rather than symptoms.

3. Temporal leakage

Using future data to predict the past. Example: training a fraud model on aggregated customer statistics that include transactions from the entire dataset, including future ones. The model learns "customers who later commit fraud" rather than "current signals of fraud."

4. Group leakage

Related samples end up in both train and test. Example: medical images where multiple scans come from the same patient. If patient A has 10 images and 5 go to train and 5 go to test, the model learns patient-specific artifacts rather than disease features.

from sklearn.model_selection import GroupKFold

# Ensure all rows from the same patient stay in the same fold
groups = df['patient_id'].values
cv = GroupKFold(n_splits=5)
scores = cross_val_score(model, X, y, groups=groups, cv=cv)

5. Preprocessing leakage

Any preprocessing step (feature selection, dimensionality reduction, outlier removal) that uses the full dataset before splitting leaks information. Always fit preprocessing onlyon the training data, and apply the fitted transformer to validation/test data.

# WRONG — feature selection sees test data
selector = SelectKBest(k=10)
X_selected = selector.fit_transform(X, y)  # leakage!
X_train, X_test = train_test_split(X_selected)

# RIGHT — pipeline ensures no leakage
from sklearn.pipeline import Pipeline

pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('selector', SelectKBest(k=10)),
    ('model', LogisticRegression())
])

cross_val_score(pipe, X, y, cv=5)  # safe: each fold fits scaler/selector on train only

Choosing the Right Metric

Accuracy is often the wrong metric. If 99% of transactions are legitimate, a model that always predicts "legitimate" achieves 99% accuracy while being completely useless.

Classification metrics

• Precision — of all predicted positives, how many are actually positive? TP / (TP + FP). High precision means few false alarms.
• Recall (Sensitivity) — of all actual positives, how many did we catch? TP / (TP + FN). High recall means few missed cases.
• F1 Score — harmonic mean of precision and recall. Useful when you need a single number and classes are imbalanced.
• AUC-ROC — area under the ROC curve. Measures ranking quality across all thresholds. Good for comparing models.
• AUC-PR (Average Precision) — area under the precision-recall curve. More informative than ROC for highly imbalanced data.

Regression metrics

• MAE (Mean Absolute Error) — average absolute difference. Robust to outliers, same units as target.
• RMSE (Root Mean Squared Error) — penalizes large errors more. Sensitive to outliers.
• MAPE (Mean Absolute Percentage Error) — percentage error. Dangerous when target values can be near zero.
• R² — proportion of variance explained. Can be negative if the model is worse than predicting the mean.

from sklearn.metrics import (
    precision_score, recall_score, f1_score,
    roc_auc_score, average_precision_score,
    mean_absolute_error, mean_squared_error, r2_score
)

# Classification
y_pred = model.predict(X_test)
y_prob = model.predict_proba(X_test)[:, 1]

print("Precision:", precision_score(y_test, y_pred))
print("Recall:", recall_score(y_test, y_pred))
print("F1:", f1_score(y_test, y_pred))
print("AUC-ROC:", roc_auc_score(y_test, y_prob))
print("AUC-PR:", average_precision_score(y_test, y_prob))

# Regression
print("MAE:", mean_absolute_error(y_test, y_pred))
print("RMSE:", mean_squared_error(y_test, y_pred, squared=False))
print("R²:", r2_score(y_test, y_pred))

AUC-PR (Average Precision) in Depth

AUC-PR, also called Average Precision (AP), measures the area under the precision-recall curve. Unlike ROC-AUC, which can look deceptively good on imbalanced data, AUC-PR focuses specifically on the positive class and is much more informative when positives are rare.

Why AUC-PR beats ROC-AUC for imbalance

ROC-AUC plots True Positive Rate vs. False Positive Rate. When negatives dominate (e.g. 99:1 ratio), even a large number of false positives barely moves the FPR. A useless model can achieve ROC-AUC > 0.9 simply by correctly ranking the abundant negatives. AUC-PR has no such blind spot: it directly measures precision at every recall level, so a useless model scores near the baseline positive rate.

How Average Precision is computed

AP is a weighted sum of precision values at each threshold, where the weight is the increase in recall from the previous threshold. Intuitively, it rewards models that maintain high precision as recall increases.

# AUC-PR is especially revealing on imbalanced data
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, roc_auc_score

# 99% negatives, 1% positives
X, y = make_classification(n_samples=10000, weights=[0.99, 0.01], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y)

model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)
y_prob = model.predict_proba(X_test)[:, 1]

print("ROC-AUC:", roc_auc_score(y_test, y_prob))       # often 0.90+
print("AUC-PR :", average_precision_score(y_test, y_prob))  # reveals true performance

Interpreting the baseline

The baseline AUC-PR for a random classifier equals the positive class prevalence. If fraud is 0.3% of transactions, a random model scores ~0.003. This makes AUC-PR easy to interpret: a score of 0.30 means the model is 100× better than random. ROC-AUC baseline is always 0.5 regardless of class balance, making it harder to gauge improvement on imbalanced data.

Resampling for Imbalanced Data

When one class heavily outnumbers the other, models tend to predict the majority class overwhelmingly. Resampling adjusts the class distribution in the training data so the model pays more attention to the minority class. Resampling does not change the test set — you always evaluate on the original distribution.

Oversampling: SMOTE

SMOTE (Synthetic Minority Over-sampling Technique) creates synthetic minority examples by interpolating between existing minority samples and their nearest neighbors. Unlike naive duplication, SMOTE expands the decision boundary rather than memorizing existing points.

from imblearn.over_sampling import SMOTE
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y)

# SMOTE only on training data — never on test data!
smote = SMOTE(random_state=42)
X_resampled, y_resampled = smote.fit_resample(X_train, y_train)

print(f"Before SMOTE: {dict(zip(*np.unique(y_train, return_counts=True)))}")
print(f"After SMOTE : {dict(zip(*np.unique(y_resampled, return_counts=True)))}")

model = RandomForestClassifier(random_state=42)
model.fit(X_resampled, y_resampled)

Undersampling

Random undersampling drops majority-class examples until classes are balanced. It is fast and reduces training time, but discards potentially useful data.Tomek links and Edited Nearest Neighbors (ENN) are smarter undersamplers that remove noisy or borderline majority samples rather than random ones.

from imblearn.under_sampling import RandomUnderSampler, TomekLinks

# Random undersampling
rus = RandomUnderSampler(random_state=42)
X_rus, y_rus = rus.fit_resample(X_train, y_train)

# Tomek links — removes pairs of nearest neighbors from different classes
# (usually majority samples that are "too close" to minority samples)
tomek = TomekLinks()
X_tomek, y_tomek = tomek.fit_resample(X_train, y_train)

Hybrid: SMOTE + Tomek / SMOTE + ENN

SMOTE alone can create noisy synthetic samples. Combining SMOTE with a cleaning step (Tomek links or ENN) often outperforms either technique alone: SMOTE expands the minority region, then the cleaner removes ambiguous borderline samples.

from imblearn.combine import SMOTETomek, SMOTEENN

smote_tomek = SMOTETomek(random_state=42)
X_st, y_st = smote_tomek.fit_resample(X_train, y_train)

Resampling inside cross-validation

Applying SMOTE before splitting creates leakage: synthetic samples may be too similar to validation data. Always resample inside each CV fold, using a pipeline.

from imblearn.pipeline import Pipeline as ImbPipeline
from imblearn.over_sampling import SMOTE
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier

# Pipeline: scale → SMOTE → train — all fitted only on training fold
pipe = ImbPipeline([
    ('scaler', StandardScaler()),
    ('smote', SMOTE(random_state=42)),
    ('model', RandomForestClassifier(random_state=42))
])

# Safe: SMOTE is applied independently inside each training fold
cross_val_score(pipe, X, y, cv=StratifiedKFold(5), scoring='average_precision')

Cost-Sensitive Learning

Not all errors are equal. In fraud detection, missing a fraud case (false negative) costs far more than falsely flagging a legitimate transaction (false positive). In medical diagnosis, a missed cancer case is catastrophic; an unnecessary biopsy is merely expensive.Cost-sensitive learning incorporates these asymmetric costs into the training process.

Class weights

The simplest approach is to assign different weights to each class during training. The model is penalized more for misclassifying high-cost classes. Most scikit-learn classifiers support a class_weight parameter.

from sklearn.ensemble import RandomForestClassifier

# "balanced" automatically sets weights inversely proportional to class frequencies
model = RandomForestClassifier(class_weight='balanced', random_state=42)

# Manual weights: e.g. false negatives cost 10x more than false positives
model = RandomForestClassifier(
    class_weight={0: 1, 1: 10},  # majority:minority
    random_state=42
)

How class_weight works

Class weights multiply the loss contribution of each sample. A weight of 10 on the minority class means each minority misclassification counts as 10 errors. Mathematically, for weighted cross-entropy:

L = -Σ w_i · [y_i · log(ŷ_i) + (1-y_i) · log(1-ŷ_i)]

Where w_i is the weight for the true class of sample i. This is mathematically equivalent to oversampling the minority class, but more memory-efficient.

Threshold tuning for business cost

Even with balanced training, the default 0.5 probability threshold is rarely optimal. If false negatives cost 10× more than false positives, you should lower the threshold to catch more positives, accepting more false alarms in exchange.

import numpy as np
from sklearn.metrics import confusion_matrix

# Define business costs
cost_fn = 100   # cost of missing a fraud
cost_fp = 10    # cost of false alarm

y_prob = model.predict_proba(X_test)[:, 1]

# Grid search the optimal threshold
thresholds = np.linspace(0.01, 0.99, 99)
best_cost, best_thresh = float('inf'), 0.5

for t in thresholds:
    y_pred_t = (y_prob >= t).astype(int)
    tn, fp, fn, tp = confusion_matrix(y_test, y_pred_t).ravel()
    total_cost = fn * cost_fn + fp * cost_fp
    if total_cost < best_cost:
        best_cost, best_thresh = total_cost, t

print(f"Optimal threshold: {best_thresh:.3f}, total cost: ${best_cost:,}")

Cost-sensitive learning vs. resampling

• Class weights — no data augmentation, faster, mathematically equivalent to oversampling, works with any classifier
• Resampling — physically changes the dataset, can help with extremely rare classes, introduces synthetic data that may not generalize

Start with class_weight='balanced' — it is free, fast, and often surprisingly effective. Move to SMOTE or other resampling only if the weighted model underperforms and you have verified no leakage.

Cost-sensitive ensembles

Advanced techniques embed costs deeper into the algorithm:

• Cost-sensitive decision trees — split criteria use misclassification cost rather than entropy/Gini
• AdaCost — boosting that increases weights of high-cost misclassifications more aggressively
• MetaCost — wraps any classifier, relabels training data based on Bayesian minimum-risk decisions

The Validation Checklist

Before declaring a model "production ready," run through this checklist:

• Did you split data before any preprocessing?
• Are your validation folds representative of real-world data?
• Did you check for duplicate rows across train and test?
• Are all features available at prediction time (no target leakage)?
• For time-series: is validation always chronologically after training?
• For grouped data: are groups kept intact across folds?
• Did you report confidence intervals (e.g. std across folds), not just a single number?
• Is your metric aligned with the business problem (not just accuracy)?
• Did you test on a truly held-out set that was never used for any decision?