Day 5 - Week 5 Comprehensive: Model Development 2 Review

📌 Key Takeaways

• The principle running through Week 5 — go down only as far as you need, and pick the tool and metric that match the symptom.
• Custom training: script mode → extended requirements → BYOC. If the question says "minimum effort," go toward the top; if it says "system dependencies," go to BYOC.
• Distributed training: slow → data parallel (SMDDP) / OOM → model parallel or sharding (SMP). Solving OOM by adding GPUs is the wrong answer.
• Observation: model anomaly → Debugger (+ the StopTraining action) / resource inefficiency → Profiler / general alarms → CloudWatch.
• Evaluation: with imbalanced data, never use accuracy (use F1, recall, AUC-PR); if large errors matter, use RMSE; a train-validation gap means overfitting.

Week 5 at a Glance

This week was the second half of model development: "train with my own code, train at scale, look inside the training, and evaluate the results properly."

[Custom Training]      How far do I bring my own code?
   Script mode → requirements extension → BYOC

[Distributed Training] What do I split?
   Data parallel (slow) ↔ Model parallel (doesn't fit)

[Debugging·Profiling]  Look inside the training
   Debugger (model tensors) ↔ Profiler (resources)

[Model Evaluation]     Is the result really good?
   Metric selection + overfitting diagnosis + cross-validation

• For custom training, make the "minimum effort" side (script mode) the default and go down only as far as needed.
• Distributed training splits by symptom (slow vs OOM), observation by concern (model vs resources), and evaluation by the nature of the problem (imbalance, outliers).

Seen Inside One Training Job

This week's four topics are not separate stories — they are different cross-sections of the same training job.

                       ┌── Custom training: what do we run it with? (container, code)
                       │
   S3 data ──► [ SageMaker training job ] ──► model.tar.gz ──► (next week) deployment
                       │        │
                       │        └── Observation: Debugger (tensors) / Profiler (resources) / CloudWatch
                       │
                       └── Distributed training: how many machines, split how?
 
                            And once it finishes ──► Evaluation: metric choice · overfitting · cross-validation

Custom Training Review

This is a spectrum of "how far do I bring my own code."

Conventions to remember: data comes in via SM_CHANNEL_<channel name>, and the model is saved to SM_MODEL_DIR (/opt/ml/model). BYOC uses /opt/ml/input/data/<channel name>/ for input and the train execution convention.

Remember the input modes too — File (copy everything, then start) / Pipe (streaming) / FastFile (lazy loading when needed). If startup delay is the problem, consider FastFile or Pipe.

💡 Related Theory: This hierarchy is separation of concerns. Training logic is separated from the execution environment: if the environment is standard you only swap the code (script mode), and you build the environment yourself only when it is special (BYOC). If the exam stresses "minimum effort," choose the upper options; if it mentions "system dependencies / unsupported framework," choose the lower ones.

Distributed Training Review

It splits according to "what you divide."

• Data parallel: Split the data across GPUs, train with model replicas, then average gradients with all-reduce. The problem it solves = speed (throughput). SMDDP. Caution: the effective batch size grows → adjust the learning rate.
• Model parallel: Split or shard the model itself across GPUs. The problem it solves = OOM (the model doesn't fit). SMP.
• Infrastructure: Inter-node communication = EFA, data supply = FSx for Lustre.

If the OOM is only "slightly short," try mixed precision, a smaller batch + gradient accumulation, and activation checkpointing before parallelizing.

⚠️ Pitfall: Any option that tries to solve OOM with data parallel (adding GPUs) is wrong. Data parallel keeps a full replica on every GPU, so if the model doesn't fit, adding GPUs just produces the same OOM. OOM is solved with model parallel/sharding. Conversely, "it's slow" means data parallel.

Debugging·Profiling Review

You look inside training from two angles.

Points to remember: Debugger rules by default only detect and report; to stop automatically you must attach an action (StopTraining). Low GPU utilization is solved in the data pipeline (more workers, FSx cache, prefetching).

The exam asks directly for the practice of linking symptom to tool and action in one line.

Debugger looks at the internal state of the mathematical process that is the model, and Profiler looks at the usage pattern of the physical resources that process runs on. You need to look at both layers to solve "why isn't training working" and "why is it expensive" at the same time.

Model Evaluation Review

You measure "generalization" with "the metric that fits the problem."

• Classification: Confusion matrix → precision (suppresses false alarms), recall (prevents misses), F1, AUC. With imbalance, never use accuracy; use F1, recall, AUC-PR.
• Regression: RMSE (sensitive to large errors), MAE (insensitive to outliers), R².
• Overfitting: Diagnose by the train-validation gap → regularization, dropout, augmentation, early stopping.
• Cross-validation: k-fold lowers the variance of the evaluation. Imbalanced = stratified, time series = time-series split.
• Leakage check: Preprocessing must always be done after the split, based on the training set. If the score is unrealistically high, suspect leakage before overfitting.

📚 Case: A team boasted of a fraud detection model with "98% accuracy," but it actually caught almost no fraud (imbalance). Its recall turned out to be 12%. After switching the evaluation metric to F1 and recall and validating with stratified k-fold, they adjusted the threshold and class weights and raised recall. Lesson: don't be fooled by a single number (accuracy); look at the metric that matches the nature of the problem.

Comprehensive Decision Flow

Tying the whole week into a single decision tree:

When training
 ├─ What kind of code?  Standard framework → script mode
 │                      +Python packages → requirements
 │                      +System packages → BYOC
 ├─ Scale?              Slow → data parallel / OOM → model parallel
 └─ When problems arise Model anomaly → Debugger / resource inefficiency → Profiler

When evaluating
 ├─ Classification?  Imbalanced → F1·recall·AUC / balanced → accuracy is OK
 ├─ Regression?      Large errors matter → RMSE / insensitive to outliers → MAE
 ├─ Generalization?  Large train-validation gap = overfitting → regularization·augmentation·early stopping
 └─ Little data?     k-fold cross-validation (imbalanced = stratified, time series = time-split)

Revisiting It in One Piece of Code

The settings from this week actually all attach to a single Estimator. Read it while checking which topic each argument came from.

from sagemaker.pytorch import PyTorch
from sagemaker.debugger import Rule, rule_configs
 
estimator = PyTorch(
    entry_point='train.py', source_dir='src', role=role,   # ← custom training (script mode)
    framework_version='2.0', py_version='py310',
    instance_type='ml.p4d.24xlarge', instance_count=2,
    distribution={'torch_distributed': {'enabled': True}},  # ← distributed training
    rules=[Rule.sagemaker(                                  # ← observation (Debugger)
        rule_configs.loss_not_decreasing(),
        actions=rule_configs.ActionList(rule_configs.StopTraining())
    )],
    metric_definitions=[                                    # ← observation (CloudWatch metric)
        {'Name': 'validation:f1', 'Regex': r'val_f1: ([0-9\.]+)'}
    ],
    enable_sagemaker_metrics=True,
    use_spot_instances=True, max_run=86400, max_wait=172800,
    checkpoint_s3_uri='s3://my-bucket/ckpt/',               # ← cost and survivability
)
estimator.fit({'training': 's3://my-bucket/train/'})

• If you publish val_f1 as a metric, the metric you chose in evaluation becomes the monitoring and tuning objective as is.
• Spot and checkpoints are a set, and rules and actions are a set. Turning on only one side has no effect.

Keyword → Answer Quick Cards

Common Wrong-Answer Patterns

Next week covers model deployment and inference — putting the trained model into an actual service.

📖 Glossary

• Script mode: A method where you hand over only the training script and leave the execution environment to the AWS container. This week's default.
• BYOC: A method where you build your own Docker image and push it to ECR. Only when system packages or a special runtime are needed.
• SM_MODEL_DIR: /opt/ml/model inside the container. Only what is saved here is uploaded to S3.
• Data parallel / model parallel: A method that splits the data to go faster / a method that splits the model to make it fit in memory.
• Sharding: A technique that stores parameters, gradients, and optimizer state spread across GPUs to reduce memory.
• EFA / FSx for Lustre: A low-latency inter-node network / a parallel file system that feeds large data quickly.
• Debugger action / Profiler: A Debugger action is the step taken on a rule violation, and StopTraining cuts off training that is leaking cost. Profiler is a tool that measures how busy the GPU, CPU, and I/O each are and finds bottlenecks.
• metric_definitions: A setting that extracts numbers from training logs with a regex and publishes them as CloudWatch metrics.
• AUC-PR: The area under the precision-recall curve. A more honest metric than ROC-AUC on data where positives are very rare.
• Data leakage: The phenomenon where information that should be unknowable at evaluation time gets mixed into training and inflates the score.