Day 3 - Overfitting/Underfitting: Diagnosis and Regularization/Data Augmentation

📌 Key Takeaways

• Even if tuning gives you good hyperparameters, a model that fits the training set well but validates poorly is useless. Generalization failure goes in two directions: overfitting and underfitting.
• High bias = underfitting (both training and validation are bad), high variance = overfitting (only training is good, with a large gap to validation). You diagnose with learning curves.
• Overfitting remedies: more data, stronger regularization, a simpler model, fewer features, early stopping, dropout, augmentation.
• Underfitting remedies: more complexity, more features, longer training, weaker regularization. More epochs or a bigger model must never be used against overfitting.
• L1 drives some weights to exactly 0 (feature selection, sparsity), L2 makes all weights small (smooth shrinkage). Scalers and encoders must be fit on the training set only to avoid leakage.

Bias and Variance, Generalization Error

Generalization error decomposes into bias and variance. The exam asks you to tell these two axes apart more than it asks about code.

Underfitting:  train error ↑  valid error ↑   (both bad, small gap)
Ideal:         train error ↓  valid error ↓   (both good)
Overfitting:   train error ↓  valid error ↑   (the gap is large)

The bias-variance tradeoff is the fundamental tension between model complexity and generalization. Raising complexity lowers bias but increases variance; lowering it does the opposite. The goal is to find the point where the sum (bias + variance) is minimal. Regularization reduces variance to fight overfitting, while adding model capacity or features reduces bias to fight underfitting.

Reading Learning Curves — Diagnosis

[Overfitting signals]
 loss │╲                    ← validation loss rises again after some point
      │ ╲___  valid          accuracy: train 99% / valid 72% (large gap)
      │  ╲   ╱‾‾‾            as epochs go on, only train loss keeps falling
      │   ╲_╱
      │    ╲_______ train
      └──────────────── epoch
              ↑ this is the early stopping point
 
[Underfitting signals]
 loss │‾‾‾‾‾‾‾‾‾‾‾ valid     accuracy: train 65% / valid 64% (both low and similar)
      │‾‾‾‾‾‾‾‾‾‾‾ train     more training leaves the loss high and flat (plateau)
      └──────────────── epoch
 
[Ideal]
 loss │╲___
      │ ╲___╲______ valid    both curves fall together and end close to each other at a low value
      │  ╲_________ train
      └──────────────── epoch

Diagnosis → Remedy Mapping

⚠️ Trap: Any option that answers overfitting with "double the epochs / stack more layers" is wrong. Both increase variance even more. Likewise, "merge the validation set into the training set" is wrong because it removes the ability to measure generalization at all.

Regularization 1: L1, L2, ElasticNet

Add a weight-magnitude penalty to the loss function to reduce variance.

# Example: L1/L2 strength in Linear Learner
hyperparameters = {
    "l1": 0.01,                  # L1 strength (sparsity)
    "wd": 0.001,                 # weight decay = L2
}

Discriminating signals:

• "There are too many features; I want unnecessary ones discarded automatically and a sparse model" → L1
• "I want to keep every feature but shrink the weights to curb overfitting" → L2

Regularization 2: Dropout, Early Stopping

These are the two techniques used mainly in deep learning.

• Dropout: Randomly deactivates neurons at each step (e.g., rate 0.5). It prevents reliance on specific neurons and gives an implicit ensemble effect. It is turned off at inference, with the weights adjusted to compensate.
• Early Stopping: Stops training when validation loss no longer improves. It cuts training off before reaching the point of overfitting.

# Example: neural network hyperparameters
hyperparameters = {
    "dropout": 0.5,
    "early_stopping_patience": 5,   # stop if no improvement for 5 epochs
}

Dropout trains a different sub-network at every step and, at inference, effectively averages them, so it works as an implicit ensemble. Early stopping halts training right before noise memorization begins, which reduces variance. Both achieve the same goal of "making the effective model simpler" in different ways.

Regularization Technique Selection Table

⚠️ Trap: Dropout and batch normalization are not "feature selection" techniques. The answer to a statement like "the model zeroes out unnecessary features by itself" is always L1.

Data Augmentation

When it is hard to collect more data, you transform the existing data to widen the training distribution. It is powerful for mitigating overfitting.

Discriminating signals:

• "Little data and overfitting / the cost of collecting more is very high" → data augmentation
• "The minority class is so rare that the model learns only the majority class" → SMOTE/oversampling (or class weights)

Data Splitting and Leakage Prevention

To measure generalization honestly, the split has to be right first.

• train / validation / test three-way split. Use the validation set to choose hyperparameters, and score the test set only once, at the very end.
• k-fold cross-validation: When data is small, average the performance so the result doesn't depend on the luck of the split.
• Leakage: When future information, features derived from the target, or test information get mixed into training, the validation score comes out unrealistically high.

Leakage Type Table

[Correct order]
Raw data
   │
 ① Split (train / validation / test)   ← from here on, every fit uses the training set only
   │
 ② Impute missing values · encode · scale  (fit on train → transform everything)
   │
 ③ Augment / SMOTE  (train only)
   │
 ④ Train → choose hyperparameters with the validation set (adjust regularization strength here)
   │
 ⑤ Score the test set once (for the final report)

What Do You Measure Generalization With — Interpreting Metrics

To look at the gap, you first need to be sure which number the gap is between. If you pick the wrong metric, the overfitting/underfitting diagnosis itself goes astray.

With imbalanced data, "train 99% / validation 98%" with a small gap must not be taken as good generalization. Both numbers may simply be pulled along by the majority class. You have to look at the confusion matrix and the minority-class Recall together.

Diagnosis and Remedy Decision Tree

Model performance falls short of expectations
├─ Is the training performance itself low?
│    ├─ Yes → underfitting (high bias)
│    │        ├─ increase model complexity (layers · tree depth · add features)
│    │        ├─ lower regularization strength (ease L1/L2/dropout)
│    │        └─ train longer (more epochs)
│    └─ No ↓
├─ Is training good but the gap to validation large?
│    ├─ Yes → overfitting (high variance)
│    │        ├─ Can you collect more data?
│    │        │    ├─ Yes → add data
│    │        │    └─ No  → data augmentation (images: rotation·crop / text: back-translation)
│    │        ├─ Too many features, with some irrelevant? → L1
│    │        ├─ Need to keep every feature and press down? → L2
│    │        ├─ Is it a neural network?                 → dropout
│    │        └─ Does validation loss rise again?        → early stopping
│    └─ No ↓
└─ Is the validation score unrealistically high?
     └─ Yes → suspect data leakage (check the fit scope of preprocessing · target-derived features · duplicates)

Question Cue → Answer Mapping

The next post covers training stability itself — batch size, learning rate schedules, vanishing/exploding gradients, and SageMaker Debugger/Profiler.

📖 Glossary

• Generalization: The property that performance holds on new data not used for training. It is the model's real goal.
• Bias / variance: The error caused by a model being too simple / how much the result swings when the training data changes only slightly. High bias means underfitting, high variance means overfitting.
• Learning curve: A graph of how training and validation loss change with epochs or data volume. The first-line tool for diagnosing overfitting and underfitting.
• Regularization: A family of techniques that add a penalty to the loss to keep the model from becoming overly complex.
• Sparsity: The state where most weights become 0 and few features are actually used. L1 induces it. wd (weight decay) is, conversely, another name for L2.
• Dropout / early stopping: A technique that randomly switches off some neurons during training to prevent reliance on specific neurons (turned off at inference) / a technique that stops training when validation loss no longer improves, so it doesn't pass the point where overfitting begins.
• Data augmentation: Creating new training samples by transforming existing data, such as image rotation or text back-translation.
• SMOTE: An oversampling technique that creates synthetic samples by interpolating between minority-class samples. Used to relieve class imbalance.
• Data leakage: The phenomenon where information that should be unknowable at training time gets mixed into the model and inflates performance. Only the validation score becomes unrealistically good.
• k-fold CV: A method that splits the data into k pieces, validates on each in turn, and averages the performance. Used when data is small and results depend on the luck of the split.