Day 2 - Distributed Training: Data Parallel and Model Parallel

When a model doesn't fit on one device or data is so large that one GPU takes days, you must split training across multiple devices/GPUs. This is distributed training. The core is "what do you split"—data or model—two branches.

In the MLA-C01 exam, distributed training appears in symptoms like "model doesn't fit GPU memory", "training is too slow", "train billions-parameter model". Today we cover data parallelism, model parallelism, and SageMaker's distributed training libraries.

Data Parallel vs Model Parallel

First clarify the two distributed paradigms: