Based on the MLS-C01 exam guideReport an error
max_wait >= max_run), and using Accuracy as the objective metric on imbalanced data.[Day1] Training execution foundation
Estimator → input mode (File/Pipe/FastFile) → distribution (data/model parallel) → Spot + checkpoints
[Day2] Hyperparameter tuning (AMT)
search space · objective metric → strategy (Bayesian/Random/Grid/Hyperband) → early stopping → warm start
[Day3] Generalization
diagnose overfitting/underfitting (bias-variance) → regularization (L1/L2/dropout/early stopping) → data augmentation → leakage prevention
[Day4] Training optimization
batch size · learning rate schedule → vanishing/exploding gradients → Debugger (quality) / Profiler (resources)💡 Related Theory: The core tension in the second half of modeling is two axes. (1) Optimization — does it reduce the loss on training data well (learning rate, batch, gradients). (2) Generalization — does it fit new data well (regularization, data, validation). These are different. If optimization succeeds (low training loss) but generalization fails (high validation loss), that is overfitting. Exam questions are almost always designed to first ask "is this symptom an optimization problem or a generalization problem?"
A training-related problem is given
├─ ① Is this a stage where you can't even run training yet? (infrastructure)
│ ├─ Data is larger than the disk / want to cut startup delay → Pipe / FastFile
│ ├─ Download everything and access randomly → File
│ ├─ Training is slow + the model fits on a GPU → data parallel
│ ├─ The model exceeds single-GPU memory → model parallel
│ └─ Cost reduction (delay tolerated) → Managed Spot + checkpoint_s3_uri
│ (max_wait >= max_run required)
├─ ② Training runs but convergence looks wrong? (optimization — Day 4)
│ ├─ NaN · divergence · oscillation → lower learning rate + gradient clipping
│ ├─ Early layers stalled · plateau → vanishing → ReLU · BN · residual · initialization
│ ├─ OOM → lower batch / gradient accumulation / mixed precision
│ └─ GPU underused · slow · expensive → diagnose the bottleneck with Profiler
├─ ③ It converges but performance is bad? (generalization — Day 3)
│ ├─ Train high, validation low, large gap → overfitting → stronger regularization · more data · augmentation · dropout · early stopping
│ ├─ Both train and validation low → underfitting → more complexity · features · longer training
│ └─ Validation score unrealistic → leakage → check fit scope · target-derived features
└─ ④ Performance is fine but you need to squeeze out more? (tuning — Day 2)
├─ Efficient search with few evaluations → Bayesian
├─ Deep learning epoch iteration + pruning → Hyperband
├─ Reduce tuning cost → early stopping (Auto) or Hyperband
└─ Reuse a previous tuning job → warm start| Cue | Choice |
|---|---|
| Data exceeds the disk, reduce startup delay | Pipe / FastFile |
| Download everything and access randomly | File |
| Speed up training, model fits on a GPU | Data parallel |
| Model exceeds single-GPU memory | Model parallel |
| Cost reduction (some delay tolerated) | Managed Spot + checkpoint_s3_uri |
Only what you write to /opt/ml/model is uploaded to S3, and Spot requires max_wait >= max_run — remember these as trap points.
| Cue | Strategy |
|---|---|
| Efficient search with few evaluations | Bayesian |
| Deep learning epoch iteration, fast pruning | Hyperband |
| Simple, fully parallel baseline | Random |
| Exhaustive search over a few categorical combinations | Grid |
| Reduce tuning cost | Automatic early stopping / Hyperband |
| Continue from a previous tuning job | Warm start |
The objective metric for imbalanced data is F1/AUC, not Accuracy. With Bayesian, raising parallelism too high lowers how much it can use earlier results.
Tuning optimizes only in the direction the objective metric points. If the metric choice is wrong, you get a well-tuned but meaningless model.
| Situation | Objective metric | Direction | Why |
|---|---|---|---|
| Balanced classification | Accuracy, AUC | Maximize | Accuracy moves honestly |
| Imbalanced classification (fraud detection, etc.) | F1, AUC, Recall | Maximize | With 1% positives, "all negative" gives 99% accuracy — Accuracy is a trap |
| Missing a case is fatal (disease, fraud) | Recall | Maximize | The cost of a false negative is greater |
| False alarms are costly | Precision | Maximize | False positives create real operational cost |
| Regression | RMSE, MAE | Minimize | Direction that reduces error |
| Custom script | metric_definitions regex required | — | The value must be parsed from the logs for AMT to compare |
| Symptom | Diagnosis | Remedy |
|---|---|---|
| Train high, validation low (large gap) | Overfitting (high variance) | Stronger regularization, more data, dropout, early stopping, augmentation |
| Both train and validation low | Underfitting (high bias) | More complexity, more features, longer training |
| Need feature selection / sparse model | — | L1 |
| Want to shrink all weights | — | L2 |
Scalers and encoders are fit on the training set only — this is the basis of leakage prevention.
| Symptom | Root cause | Response |
|---|---|---|
| NaN / divergence / oscillation | Learning rate too high, exploding gradients | Lower learning rate, gradient clipping |
| Early layers not learning, loss plateau | Vanishing gradients | ReLU, batch normalization, residual connections, initialization |
| OOM | Excessive batch size | Lower batch, gradient accumulation, mixed precision |
| Automatic detection of quality anomalies | — | Debugger |
| Resource bottleneck, GPU underuse | — | Profiler |
| Feature | What it does | When to pick it | Confusion warning |
|---|---|---|---|
| Training Job (Estimator) | Managed one-off training run | When running training as a container | The instance exists only for the duration of the job |
| Input mode File/Pipe/FastFile | How data is delivered from S3 to the container | Depending on data size and startup-delay requirements | Pipe = streaming, FastFile = lazy loading + random access |
| Data parallel / model parallel | Distribute training across multiple devices | Speed problem vs model size problem | Tell whether "GPU memory exceeded" is caused by the model or the batch |
| Managed Spot Training | Cheap, interruptible instances | Cost reduction is the goal, delay is tolerated | Without checkpoints, the savings are meaningless |
| AMT (tuning job) | Automatic hyperparameter search | When you need to find the optimal combination | Can't fix convergence problems in training itself |
| Debugger | Detects quality anomalies from tensors during training | Vanishing, exploding, overfitting, stalled loss | Not for resource analysis |
| Profiler | Analyzes CPU/GPU and I/O resource bottlenecks | GPU underuse, slowness, cost inefficiency | Not for detecting quality anomalies |
| Model Monitor | Watches drift on deployed endpoints | The operations stage, not training | Picking it for anomaly detection during training is wrong |
| Setting | Where | What happens if you miss it |
|---|---|---|
output_path / /opt/ml/model | Estimator | Only what is written to this path is uploaded to S3 automatically |
input_mode (File/Pipe/FastFile) | TrainingInput | With File, training doesn't start until the download finishes |
instance_count + distribution | Estimator | If you only add instances without choosing a distribution strategy, it won't work as intended |
use_spot_instances + max_wait | Estimator | Unless max_wait >= max_run, the Spot configuration isn't valid |
checkpoint_s3_uri | Estimator | Without it, a Spot interruption means retraining from scratch |
strategy / early_stopping_type | HyperparameterTuner | Hyperband has early stopping built in and ignores early_stopping_type |
max_jobs / max_parallel_jobs | HyperparameterTuner | With Bayesian, raising parallelism too high lowers efficiency |
objective_metric_name | HyperparameterTuner | Putting Accuracy on imbalanced data makes the tuning itself futile |
l1 / wd | Linear Learner | wd is L2 (weight decay) — only the name differs |
gradient_clipping / lr_scheduler | Training hyperparameters | First-line means of handling explosion and divergence |
Case A: A text classification model has 96% training accuracy and 75% validation accuracy. Data is scarce and collecting more is hard. You also want to cut tuning cost.
Solution:
Case B: You are training a large model on 8TB of data and it is too slow and too expensive. It also exceeds single-GPU memory.
Solution:
| Trap | Why it's wrong | Correct judgment |
|---|---|---|
| Responding to overfitting with "more epochs / bigger model" | It increases variance and makes things worse | Regularization, data, augmentation, early stopping |
| Reflex "GPU memory exceeded = model parallel" | The cause may be the batch | The model itself doesn't fit → model parallel / because of the batch → accumulation, mixed precision, lower batch |
| Confusing Debugger and Profiler | They look at different things | Quality = Debugger, resources = Profiler, post-deployment drift = Model Monitor |
| Using Spot without checkpoints | An interruption means retraining from scratch → savings vanish | Set checkpoint_s3_uri, max_wait >= max_run |
| Accuracy as the objective metric on imbalanced data | Predicting only the majority class still scores high | F1 / AUC / Recall |
| Merging the validation set into training to raise performance | Measuring generalization becomes impossible | Validation is for selection, test is scored once at the end |
| Fitting a scaler on the whole dataset | Test information leaks into training | fit on the training set only, transform for the rest |
| Setting maximum parallelism for Bayesian | It decides the next trial without seeing earlier results | Moderate parallelism; fully parallel suits Random |
| Question cue | Answer to pick | Which Day |
|---|---|---|
| "Training loss is very low but validation loss is much higher" | Optimization succeeded, generalization failed = overfitting | Day 3 |
| "Both training and validation are low and the gap is small" | Underfitting → more complexity · features · training | Day 3 |
| "The data is larger than the instance disk" | Pipe / FastFile input mode | Day 1 |
| "The model exceeds single-GPU memory" | Model parallel | Day 1 |
| "Cut cost substantially but delay is tolerated" | Managed Spot + checkpoints | Day 1 |
| "Quickly discard hopeless candidates to cut tuning cost" | Hyperband | Day 2 |
| "Search efficiently with few training runs" | Bayesian | Day 2 |
| "Continue exploring from last tuning job's results" | Warm start | Day 2 |
| "Objective metric for data with 1% positives, like fraud detection" | F1 / AUC (Accuracy is a trap) | Day 2 |
| "Automatically zero out features to get a sparse model" | L1 | Day 3 |
| "Labeling cost is high so you can't grow the data" | Data augmentation | Day 3 |
| "Loss suddenly diverges to NaN" | Lower learning rate + gradient clipping | Day 4 |
| "GPU utilization 30%, suspect data-loading bottleneck" | Profiler | Day 4 |
| "Automatically detect and alert on vanishing, overfitting, non-decreasing loss" | Debugger | Day 4 |
| "Monitor input distribution changes after deployment" | Model Monitor (next week's scope) | Week 9 |
Next week moves to Domain 4 and covers how to deploy, operate, and monitor models.
Click a choice to reveal the answer and explanation.
Question 1
A model's training loss is very low, but its validation loss is much higher. Which statement BEST characterizes this situation?
Question 2
The 8 TB of training data is larger than the instance disk, the model exceeds a single GPU's memory, and you also want to reduce cost. Which combination is MOST appropriate?
Question 3
In deep learning tuning, you want to converge quickly without wasting resources on unpromising candidates while reducing tuning cost. Which AMT strategy is MOST suitable?
Question 4
During training, the loss suddenly diverged to NaN. Which of the following should be checked and applied first?
Question 5
Training converges normally, but GPU utilization is low at 30% and you suspect a data loading bottleneck. You want to analyze resource utilization to improve cost and speed. Which tool is MOST suitable?