End-to-End ML Project

1. Problem & Goal

Score every online card transaction in real time and decide whether toapprove, challenge, ordecline it. The system must minimize fraud losses while keeping false declines (good customers wrongly blocked) low and responding in under 100 ms.

• Business metric: net fraud $ saved minus false-decline cost.
• Model metric: AUC-PR, plus recall at a fixed precision.
• System metric: p99 latency < 100 ms, 99.99% availability.

2. Data

Inputs come from multiple sources, joined on transaction_id anduser_id:

• Transaction logs (amount, merchant, MCC, currency, timestamp)
• User profile & tenure, historical spend patterns
• Device fingerprints, IP, geolocation
• Behavioral sequences (page views, time on checkout)
• Chargeback labels (the ground truth, available with 30–90 day lag)

Positive class (fraud) is typically 1–5% — heavily imbalanced. Labels arrive late, so we train on a delayed but high-quality label set.

Feature engineering

• Velocity: # transactions / amount in last 1m, 1h, 24h per user, card, device, IP.
• Match features: billing country vs IP country, device-user history.
• Risk encodings: merchant risk score, MCC fraud rate (target-encoded with smoothing).
• Temporal: time-of-day, day-of-week, time since last transaction.

3. Modeling Approach

Gradient-boosted trees (XGBoost / LightGBM) are the workhorse for tabular fraud — strong baselines, calibrated probabilities, and fast inference. Often paired with an unsupervised anomaly score (Isolation Forest or autoencoder reconstruction error) as an extra feature.

import lightgbm as lgb

model = lgb.LGBMClassifier(
    n_estimators=2000,
    learning_rate=0.03,
    num_leaves=63,
    min_child_samples=200,
    subsample=0.8,
    colsample_bytree=0.8,
    objective="binary",
    scale_pos_weight=25,        # handle class imbalance
    metric="average_precision",
)

model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    callbacks=[lgb.early_stopping(100)],
)

Validation

• Time-based split: never randomly shuffle — fraud patterns drift.
• Out-of-time test set: last 4 weeks held out, untouched.
• Threshold selection: pick score cutoff on the precision-recall curve to hit a target review rate.

4. System Architecture

Client / Checkout
       │
       ▼
[ API Gateway ] ──► [ Feature Service ] ──► [ Online Store (Redis) ]
       │                       │
       │                       └──► [ Offline Store (S3 / BigQuery) ]
       ▼
[ Scoring Service ] ──► [ Model Artifact ] ──► decision
       │
       ▼
[ Decision Engine ]  ──► approve / step-up / decline
       │
       ▼
[ Event Bus (Kafka) ] ──► training data, monitoring, analytics

• Feature store: Redis online for <10 ms reads; S3/BigQuery offline for training. Same code path for both (Feast or in-house).
• Scoring service: Python FastAPI or Go, model loaded into memory, autoscaled.
• Streaming: Kafka topics for transactions, decisions, and feedback.

5. Model Development Lifecycle

• EDA → feature engineering → baseline (logistic regression) → boosted trees.
• Hyperparameter search with Optuna, tracked in MLflow / Weights & Biases.
• Bias & fairness checks across geography, card type, customer segment.
• Model card documenting assumptions, training data window, known limitations.

6. Deployment & MLOps

• CI builds a Docker image with the model artifact baked in (or pulled from a registry).
• Canary deploy 5% → 25% → 100%, with automatic rollback on metric regression.
• Shadow mode for new models — score in parallel without affecting decisions.
• Scheduled retraining weekly; triggered retraining on drift alerts.

# Simplified k8s deployment for the scoring service
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fraud-scorer
spec:
  replicas: 6
  template:
    spec:
      containers:
        - name: scorer
          image: registry/fraud-scorer:v1.42.0
          resources:
            requests: { cpu: "1", memory: "2Gi" }
            limits:   { cpu: "2", memory: "4Gi" }
          readinessProbe:
            httpGet: { path: /healthz, port: 8080 }

7. Monitoring

• Data drift: PSI / KS on input features vs training distribution.
• Prediction drift: score distribution & approval rate over time.
• Performance: AUC-PR on labels as they mature; alert if it drops.
• Operational: p50/p99 latency, error rate, queue depth.
• Business: $ fraud caught, false decline rate, manual review backlog.

8. Business Impact

A well-tuned system typically delivers:

• 20–40% reduction in fraud losses
• 15–30% fewer false declines than rules-only baselines
• Faster checkouts for trusted users (fewer step-up challenges)
• Lower manual review cost via better score calibration

9. Worked Example — "PayFast" Checkout Fraud

Let's run the full project on a concrete (fictional) merchant called PayFast, an e-commerce platform processing card-not-present transactions.

9.1 Assumptions

• Volume: 1,000,000 transactions / month, average ticket $80.
• Base fraud rate: 1.2% of transactions (12,000 / month), average fraud amount $220.
• Current baseline: hand-written rules — catches 55% of fraud, false-decline rate 1.8%.
• Economics: each missed fraud costs $220 (chargeback + fee); each false decline costs $12 in lost margin & lifetime value.
• Latency budget: 80 ms p99 from request to decision.
• Label lag: 45 days for chargebacks to mature.

9.2 Dataset

• 18 months of transactions: ~18M rows.
• Training window: months 1–15. Validation: month 16. Out-of-time test: months 17–18.
• 137 engineered features (velocity, device, geo-mismatch, merchant risk, behavioral).
• Positive class: 1.2% — handled with scale_pos_weight=80 in LightGBM.

9.3 Models compared

LightGBM is chosen: nearly the best quality, well under the 80 ms budget, and simpler to operate than the stacked autoencoder variant.

9.4 Threshold selection

On the out-of-time test set, we sweep the decision threshold and pick the operating point that maximizes net business value:

Net value = (TP * $220)  -  (FP * $12)  -  (FN * $220)

Threshold   Precision   Recall   TP     FP     FN    Net $ / month
  0.30        0.42       0.86   10,320  14,250  1,680  $1,857,000
  0.45        0.58       0.78    9,360   6,780  2,640  $1,396,440  ← chosen
  0.60        0.71       0.66    7,920   3,230  4,080  $861,720

We pick 0.45: slightly lower net $ than 0.30, but the false-decline rate (0.68%) keeps customer-experience SLAs intact and the manual-review team can handle the volume.

9.5 Results after 90 days in production