Day 3 - Data Leakage: Causes, Detection, Prevention, and Time-Series and Target Leakage

📌 Key Takeaways

• Leakage is information that would not have been known at prediction time getting into training. Only the validation score inflates, and the model collapses in real production.
• Target leakage has outcome information baked into the feature itself, so no matter how well you split, it is not caught. You must examine when each feature is generated.
• Time-series leakage comes from random splits and aggregation over future windows. Prevent it with chronological splits and past-only aggregation (rolling after shift).
• Train-test contamination arises from preprocessing before the split. Split first and fit only on the training part, and enforce this with a Pipeline.
• The diagnostic question is always one: "At the very moment of prediction, could we really have known this value?" If not, it is leakage.

What Is Data Leakage?

The validation score is perfect, yet the model fails miserably in real production. The culprit is almost always one thing: data leakage. Leakage is the phenomenon where a model gains access to information it could not actually have had at training time, so the evaluation score is artificially inflated.

MLS-C01 is obsessed with leakage. The answer to a scenario like "99% on validation but 60% in production?" is almost always leakage.

The key question is always the same. "At the moment the prediction is made, could we really have known this value?" If not, it is leakage.

Leakage is essentially the same as cheating on an exam: the model has already seen the answer sheet. Statistically, it violates the independence assumption of the validation set. For a validation score to estimate generalization performance on unseen data, the validation set must be completely independent of the training process. With leakage, the validation score measures not generalization but "the ability to reproduce information already seen," and every model-selection and hyperparameter-tuning decision made on top of it is contaminated.

Leakage Type → Symptom → Detection → Prevention (Key Table)

Target Leakage: Future and Outcome Information Gets Mixed In

Target leakage occurs when information about the target being predicted is included in the features. It is the most insidious and the most common.

Signs of target leakage:

• A single feature's correlation or importance is abnormally high (one variable explains 95% of the accuracy).
• The feature is created at the same time as or after the target.
• The answer to "Can we really know this value at prediction time?" is "No."

# Check for suspected target leakage: does a single feature explain the target too well?
import pandas as pd
 
corr_with_target = (
    df.corr(numeric_only=True)["churned"]
    .abs()
    .sort_values(ascending=False)
)
print(corr_with_target.head(10))
# If any feature's correlation is 0.95 or higher, first ask "Is this a value we could know at prediction time?"

⚠️ Pitfall: Target leakage is not caught even if you split train/validation/test perfectly, because the leakage is baked into the feature itself. Every split is contaminated equally. So splitting alone is not enough; you must use domain knowledge to examine when each feature is generated (timestamp) and how that relates to the prediction time. "Validation is high and test is high, but only production is low?" → Suspect target leakage first.

Leakage Suspicion Signal Table

Time-Series Leakage: The Future Seeps into the Past

The most common leakage in time-series data is predicting the past with future information.

Causes:

• Random split: Shuffling a time series puts future data in training and past data in test → predicting the past with the future
• Window aggregation that includes the future: Computing a "30-day average" that includes data after the prediction time
• Using global statistics: Normalizing with the overall mean → future information seeps into past rows
• lag/rolling boundary errors: A wrong shift puts the t+1 value into time t

[Wrong random split]                    [Correct chronological split]
Time axis ───────────────────▶          Time axis ───────────────────▶
 T T V T V T T V T V T T V T             T T T T T T T T | V V V | E E
 └─ training (T) and validation (V)       training          validation  evaluation
    are mixed                             └── always from past → future ──┘
    → model sees t+1 and "predicts" t     (forward-chaining, same as the deployment situation)
    → training backward in time = leakage

import pandas as pd
 
df = df.sort_values("date")
 
# Correct approach: rolling that looks only at the past (shift excludes the current row)
df["sales_ma7"] = df["sales"].shift(1).rolling(7).mean()  # t-1 ~ t-7
 
# Split time series chronologically (no random split)
cutoff = "2025-01-01"
train = df[df["date"] < cutoff]
test = df[df["date"] >= cutoff]

The reason a random split is fatal for time series is temporal dependence. Ordinary data assumes rows are independent, but a time series has strong correlation between adjacent points in time. If you shuffle randomly, the model sees the pattern at t+1 and "predicts" t, learning a backward-in-time relationship that is impossible in reality. So time-series validation must always follow the principle "train on the past and evaluate on the future (forward-chaining)," which leads into Day 4's time-series splitting.

Train-Test Contamination: Leakage Through Preprocessing

Even if you split properly, if you preprocess on the entire dataset before splitting, test information leaks into training.

The solution is a Pipeline that confines preprocessing inside the split and cross-validation.

[Order that leaks]                     [How a Pipeline prevents it]
Entire dataset                          Entire dataset
   │                                       │
   ▼ fit(scaler) ← includes test stats     ▼ split
Scale everything                        training fold │ validation fold
   │                                       │         │
   ▼ split                            fit+transform  transform only
training │ test                            │         │  (uses training fold statistics only)
   └─ already contaminated                 └─ train model ▶ evaluate

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
 
# Pipeline: fit happens only on each fold's training part, blocking leakage
pipe = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
    ("clf", LogisticRegression()),
])
 
# Each fold's preprocessing is fit on the training part only
scores = cross_val_score(pipe, X, y, cv=5)

⚠️ Pitfall: "Can't I scale on the entire dataset and then split?" is an easy question to get confused about. No, you can't. If StandardScaler is fit on the entire dataset, test statistics leak into the training scale and the validation score is optimistically inflated. You must fit only on the training part and apply only transform to the test. In cross-validation, a Pipeline enforces this rule automatically for every fold.

Leakage and Metrics: Why a "Too-Good Score" Is a Warning Sign

High accuracy on imbalanced data may be majority-class bias, not leakage. To tell the two causes apart, you must look at minority-class metrics (recall, PR-AUC). Both concluding "it's leakage" from accuracy alone and feeling reassured that "it's fine" are wrong.

Data Leakage Prevention Checklist

A consistent procedure for preventing leakage in practice:

1. Split first: Do the train/test split before any preprocessing
2. Fit on the training part only: fit scalers, encoders, and imputers on the training part only
3. Use a Pipeline: Confine preprocessing inside the cross-validation loop
4. Split time series chronologically: No random splits; lag/rolling reference only the past
5. Audit feature generation time: Confirm that each feature is a value knowable at prediction time
6. Be suspicious of high validation scores: If the score is too good, suspect leakage first
7. Clean up groups and duplicates: Handle the same user and duplicate records before splitting

AWS Service Mapping Table

Question Clue → Answer Mapping Table

The next article is the practical companion to these principles: validation design: splitting, cross-validation, stratification, and time-series splits.

📖 Glossary

• Data leakage: An accident in which information that would not have been known at prediction time flows into training and inflates performance.
• Target leakage: Leakage in which outcome information about the prediction target is contained in a feature. It is not caught by splitting.
• Train-test contamination: The phenomenon in which preprocessing the entire dataset before splitting lets test information leak into training.
• Time-series leakage: Leakage in which future data is used to predict the past. Random splitting is the main culprit.
• Group leakage: The phenomenon in which rows from the same user or patient are split between training and validation, so performance is overestimated.
• forward-chaining: A time-series validation method that always trains on the past and evaluates on the future.
• Rolling after shift: A technique that computes moving statistics using only past values, excluding the current row, to prevent leakage.
• Pipeline: A structure that bundles preprocessing and the model together and forces fit to occur only on each fold's training part.
• fit / transform distinction: The step that learns statistics (fit) versus the step that applies them (transform). Use only transform on the test set.
• Validation set independence: The principle that for a validation score to estimate generalization performance, the validation data must be completely separated from the training process.