Day 4 - Validation Design: Train/Validation/Test Splits, Cross-Validation, Time-Series Splits, Stratified Sampling

📌 Key Takeaways

• Keep the test set sealed until every decision is made and look at it exactly once. The moment you look and fix, the test set becomes a validation set.
• When data is scarce, use k-fold cross-validation so that every row is used for validation, and look at the mean ± standard deviation of the k scores (k=5 or 10 is the balance point).
• For classification, especially with class imbalance, stratification is practically mandatory. It preserves the original class ratio in every split.
• For time series, use TimeSeriesSplit. The training period must always be earlier than the validation period (forward-chaining).
• When the same user or patient has multiple rows, you must split at the group level with GroupKFold so that performance is not inflated.

Why Split in Three: train / validation / test

Yesterday we saw how leakage ruins validation. Today is its mirror image: designing validation without leakage so you honestly estimate a model's generalization performance. Good model selection is ultimately the ability to credibly estimate "how well will it do on data it has never seen?", and that credibility depends entirely on the validation design.

The most basic step is to split the data into three. Each has a different role.

Key principle: keep the test set sealed until every decision is made, and look at it exactly once. If you modify the model after seeing the test score, the test set effectively becomes a validation set and the final estimate is contaminated optimistically.

from sklearn.model_selection import train_test_split
 
# Step 1: train+val vs. test (test is sealed)
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
# Step 2: train vs. validation
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.25, random_state=42, stratify=y_temp
)
# Result: 60% / 20% / 20%

The reason for separating the validation set from the test set is the multiple comparisons problem. If you compare dozens of hyperparameter combinations on the validation set, a combination that happens to fit that validation set well gets selected by chance, and "selection bias" accumulates in the validation score. That score no longer estimates generalization performance fairly. So after all selections are finished, you measure final performance on a test set that has never been used. Only then is the estimate honest.

Role Comparison Table for the Three Splits

k-fold Cross-Validation: Saving Data

A single split depends on luck. Especially when data is scarce, which rows land in validation swings the score greatly. k-fold cross-validation divides the data into k parts, uses each part once as validation while training on the rest, and averages the scores from the k repetitions.

5-fold cross-validation (■ = validation fold, □ = training fold)

Fold 1  ■ □ □ □ □   → score₁
Fold 2  □ ■ □ □ □   → score₂
Fold 3  □ □ ■ □ □   → score₃
Fold 4  □ □ □ ■ □   → score₄
Fold 5  □ □ □ □ ■   → score₅
                     ────────────────────
                     Report as mean ± standard deviation
                     (every row is used for validation exactly once)

Benefits of cross-validation: (1) every row is used for validation once, saving data, and (2) you obtain the mean and variance of the k scores, so you can also see the stability of performance.

from sklearn.model_selection import StratifiedKFold, cross_val_score
 
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(pipe, X, y, cv=skf, scoring="f1")
print(f"F1: {scores.mean():.3f} ± {scores.std():.3f}")  # mean ± standard deviation

Choosing k involves a bias-variance tradeoff. When k is small (e.g., 2-fold), each training set becomes small and the estimate is pessimistically biased. When k is very large (LOO, k=n), the training set is nearly the whole dataset so bias is small, but the fold results become similar to each other, so the estimate variance grows and the computation explodes. Empirically, k=5 or 10 is the balance point of bias, variance, and cost, and is widely used.

k Value Tradeoff Table

⚠️ Pitfall: "The larger k is, the always better" is a wrong answer. As k grows, computational cost increases in proportion to k, and the training sets become nearly identical so the correlation between fold scores rises, which actually increases the variance of the mean estimate.

Stratified Sampling: Preserve the Ratio

In classification with imbalanced classes (e.g., 1% fraud), a random split is risky. With bad luck, validation may contain almost no minority-class samples, or even zero. Stratified sampling preserves the original class ratio in every split.

• train_test_split(..., stratify=y): preserves the ratio when splitting
• StratifiedKFold: preserves the ratio in each fold

In classification, stratification is almost always recommended, and for imbalanced data it is practically mandatory. In regression, the target is sometimes divided into bins for stratification.

from sklearn.model_selection import train_test_split
 
# stratify=y preserves the original class ratio across the split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)
print(y_train.value_counts(normalize=True))  # check whether it matches the original ratio

⚠️ Pitfall: If you randomly split imbalanced data without stratification, some fold may have zero minority-class samples, making F1 and recall incalculable or distorted. In classification, if accuracy is high but the minority class is not caught at all, first check for a stratified split.

Metric Interpretation Table for Imbalanced Data

Even if you fix the split with stratification, choosing the wrong metric will still mislead you.

The reason accuracy is a trap on imbalanced data is that the score is high even if you only get the majority class right. Validation design (stratification) and metric choice (PR-AUC, recall) must be fixed together. Fixing only one of them still leads you to pick the wrong model.

Time-Series Splits: Never Go Against Time

As we saw on Day 3, using a random split or ordinary k-fold on a time series leaks the future into the past. Time series need a split that respects time order.

scikit-learn's TimeSeriesSplit implements this forward-chaining. In every split, training is always earlier than validation.

TimeSeriesSplit (expanding window, n_splits=4)
Time axis ───────────────────────────────────────▶

Split 1  [■■■■]───[▨▨]  ·  ·  ·  ·        ■ training  ▨ validation
Split 2  [■■■■ ■■]───[▨▨]  ·  ·
Split 3  [■■■■ ■■ ■■]───[▨▨]  ·
Split 4  [■■■■ ■■ ■■ ■■]───[▨▨]

  · The training period expands each time, and the validation period is always the "next" point in time
  · Same structure as real deployment (predicting the future with a model trained on the past)

from sklearn.model_selection import TimeSeriesSplit
 
# Chronological split: training is always earlier than validation (blocks leakage)
tscv = TimeSeriesSplit(n_splits=5)
for train_idx, test_idx in tscv.split(X):
    # every point in time in train_idx < the points in test_idx
    X_tr, X_te = X.iloc[train_idx], X.iloc[test_idx]
    # train/evaluate the model ...

Time-series validation differs from ordinary cross-validation because the row-independence assumption breaks. Ordinary k-fold assumes rows are exchangeable, but in a time series the order itself is information. TimeSeriesSplit always maintains "train on the past, evaluate on the future," simulating the real deployment situation (predicting the future with a model built on the past). Thanks to this alignment, the validation score comes to resemble real production performance.

Validation Strategy Selection Table

AWS Service Mapping Table

Question Clue → Answer Mapping Table

In the next article, we tie together the whole week, from statistics basics to validation design, into a single flow.

📖 Glossary

• Validation set: Data used repeatedly for hyperparameter tuning and model selection. It is not for reporting final performance.
• Test set: Data for final evaluation that is looked at exactly once, after all decisions are made.
• Multiple comparisons problem: The phenomenon in which comparing multiple candidates on the same validation set selects one that fits by chance, optimistically biasing the score.
• k-fold cross-validation: A method that divides the data into k parts and trains and evaluates k times, using each part once as validation.
• Stratified sampling: A way of splitting so that each split preserves the original class ratio.
• GroupKFold: Cross-validation that keeps rows from the same group (user, patient) from being separated between training and validation.
• TimeSeriesSplit: Time-series-specific cross-validation that splits so the training period is always earlier than the validation period.
• Nested CV (nested cross-validation): A method that tunes in inner folds and evaluates in outer folds, eliminating selection bias as well.
• Bias-variance tradeoff (choosing k): The relationship in which a small k biases the estimate pessimistically, while a very large k increases variance and computational cost.
• PR-AUC: The area under the precision-recall curve. Gives a more honest signal than ROC-AUC when the positive ratio is very low.