Day 4 - Learning Optimization: Batch Size, Learning Rate, Gradient Issues, Debugger/Profiler

📌 Key Takeaways

• If overfitting and underfitting were about "what the model learns," today is about "how to make it converge well." Even with the same model and data, training diverges or stalls depending on batch size, learning rate, and gradient stability.
• Loss NaN, divergence, oscillation → learning rate too high or exploding gradients. The remedy is a lower learning rate + gradient clipping.
• Early layers do not learn and loss stalls → vanishing gradients. The remedy is ReLU-family activations, batch normalization, residual connections, and proper initialization.
• OOM → reduce batch size, gradient accumulation, mixed precision. The rule of thumb is that when you increase the batch size, you increase the learning rate with it (linear scaling).
• One-line tool distinction: training quality anomalies = Debugger, resources/bottlenecks/cost = Profiler. Post-deployment drift is Model Monitor.

The Training Loop at a Glance

       ┌──────────────── 1 step ────────────────┐
Data →│ Sample batch → Forward → Compute loss → Backprop (gradients) → Update weights │→ Next step
       └──────────────────────────────────────────────┘
          ↑batch size     ↑activation fn  ↑vanishing/exploding  ↑learning rate/schedule
 
  Batch size    : Number of samples used in one update → affects memory, GPU utilization, generalization
  Learning rate : How far weights move in one step → separates divergence from stalling
  Gradient      : Multiplied back through the layers, vanishing to 0 (vanishing) or blowing up (exploding)
  1 epoch       : One full pass over the training data = (number of samples / batch size) steps

Batch Size

The number of samples used in one weight update.

Batch Size ↔ Learning Rate ↔ Generalization Trade-off

• When you increase the batch size, you usually need to increase the learning rate too, to keep the convergence speed (linear scaling rule of thumb).
• A common cause of GPU out-of-memory (OOM) is an excessive batch size → reduce the batch or use gradient accumulation.

One view is that the gradient noise of small batches helps the model escape sharp minima of the loss surface and head toward flatter regions that generalize better. Conversely, a very large batch gives accurate, stable gradients but loses this exploratory quality, so generalization can drop. That is why batch size is not merely a speed issue but an optimization-versus-generalization trade-off.

Learning Rate and Schedules

The learning rate is how far the weights move in one step. It is the most sensitive hyperparameter in training.

Too large → loss diverges or oscillates (NaN loss)
Too small → convergence is very slow and gets stuck in local minima
Right     → decreases steadily

Distinguishing signals:

• "Loss is NaN / diverges / oscillates widely" → the learning rate is too high.
• "Training barely moves / is too slow" → the learning rate is too low, or vanishing gradients.

Gradient Problems: Vanishing and Exploding

These are problems caused by gradients being multiplied during backpropagation in deep neural networks.

Symptom → Cause → Remedy Table

hyperparameters = {
    "gradient_clipping": 1.0,   # upper bound on the gradient norm
    "lr_scheduler": "cosine",
}

Sigmoid and tanh have derivatives close to 0 in their saturated regions, so in a deep network gradients converge to 0 in a chain reaction (vanishing). ReLU has a derivative of 1 in the positive region, which mitigates this. Explosion is the opposite: values greater than 1 are multiplied in a chain and diverge, and gradient clipping suppresses it by setting an upper bound on the norm. Batch normalization and residual connections are structural devices that mitigate both problems.

SageMaker Debugger

It captures tensors during training (gradients, weights, loss, etc.) and automatically detects training anomalies.

• When a rule is violated, a CloudWatch event is emitted → it can trigger automatic notification or early stopping of training.
• Tensors are stored in S3 for post-hoc analysis.

from sagemaker.debugger import Rule, rule_configs
 
rules = [
    Rule.sagemaker(rule_configs.vanishing_gradient()),
    Rule.sagemaker(rule_configs.loss_not_decreasing()),
]
est = Estimator(..., rules=rules)

Distinguishing signal: "automatically detect and alert on vanishing/exploding gradients, overfitting, and stalled loss during training" → Debugger.

SageMaker Profiler

It analyzes system resources and performance bottlenecks of training (efficiency, not model quality).

• It collects system metrics such as CPU/GPU utilization, GPU memory, I/O wait, and data loading bottlenecks.
• It produces diagnostics and recommendation reports such as "the GPU is idle (underutilized)," "data loading is the bottleneck," and "CPU-bound."
• Cost optimization: if instances are used inefficiently, it recommends smaller or fewer instances.

Comparing Observation Tools — The Confusing Three

Distinguishing signals:

• "Training quality anomalies (vanishing/exploding/overfitting)" → Debugger.
• "Resource utilization, bottlenecks, GPU underutilization, training speed/cost inefficiency" → Profiler.

⚠️ Trap: Some options mix Debugger, Profiler, and Model Monitor. Quality = Debugger / resources = Profiler / post-deployment drift = Model Monitor — this one line settles most of them.

Additional Training Optimization Techniques

⚠️ Trap: "GPU out of memory" does not automatically mean model parallelism is the answer. If the model itself does not fit, model parallelism; if it is because of the batch, gradient accumulation, mixed precision, or a smaller batch.

Diagnostic Tree When Training Is Not Going Well

Training is not going as expected
├─ Does the job fail outright?
│    └─ OOM error → memory problem
│         ├─ The model itself does not fit on a single GPU → model parallelism (Day 1)
│         └─ Because of the batch → reduce batch / gradient accumulation / mixed precision
├─ Is the loss NaN, diverging, or oscillating widely?
│    └─ Yes → learning rate too high or exploding gradients
│              → lower learning rate + gradient clipping (+ batch normalization)
├─ Is the loss barely decreasing and flat?
│    ├─ Early-layer weights barely change → vanishing gradients
│    │      → ReLU family / batch normalization / residual connections / revisit initialization
│    └─ Decreasing only slowly throughout → learning rate too low → raise learning rate, warmup
├─ Does the loss decrease well, but training is slow and expensive?
│    └─ Yes → resource problem → identify the bottleneck with Profiler
│              ├─ GPU idle / I/O wait → improve the input pipeline
│              └─ Low utilization throughout → shrink instances to save cost
└─ Does the loss decrease but validation gets worse?
     └─ Yes → generalization problem (Day 3) → regularization, augmentation, early stopping
              (can be detected automatically with Debugger's overfit / overtraining rules)

Passage Clue → Answer Mapping

Tomorrow we review all of Week 8 (training jobs, AMT, generalization, optimization) together and tie it up from the exam perspective.

📖 Glossary

• Batch size: The number of samples used when updating weights once. It affects memory, GPU utilization, and generalization at the same time.
• Learning rate: The value that determines how far the weights move in one step. The most sensitive hyperparameter in training.
• Learning rate schedule / warmup: A rule that gradually decreases the learning rate as training progresses (step, exponential, cosine) / conversely, a phase that starts very small and slowly raises it to reduce early instability.
• Vanishing gradient: A phenomenon where gradients approach 0 as they pass through layers during backpropagation, so early layers do not learn. Saturation of sigmoid and tanh is the main culprit.
• Exploding gradient: The opposite, where gradients grow in a chain and the loss diverges to NaN. It is suppressed by gradient clipping, which caps the norm.
• Residual connection / batch normalization: A structure that skips the input over several layers and adds it back so gradients flow along a shortcut (ResNet) / a technique that normalizes layer outputs per batch to stabilize training. Both mitigate vanishing and exploding gradients.
• Mixed precision / gradient accumulation: A method that mixes FP16/BF16 to cut memory and raise speed / a technique that collects gradients from small batches and updates at once to get a large-batch effect. Both are used to avoid OOM.
• SageMaker Debugger: A feature that captures tensors during training and automatically detects quality anomalies such as vanishing, exploding, and overfitting with built-in rules, raising CloudWatch events.
• SageMaker Profiler: A feature that analyzes system resources such as CPU/GPU utilization and I/O, pointing out bottlenecks and cost inefficiencies.
• OOM (Out Of Memory): A GPU out-of-memory error. It occurs when the batch is too large or the model itself does not fit on the device.