Even if data is well prepared, if the training job reads that data slowly, the GPU sits idle waiting for data. Expensive GPU instances becoming idle due to I/O bottlenecks is one of the most common and costly wastes in practice. Therefore, "how data is streamed to the training container" directly impacts performance and cost.
Today, we cover SageMaker's input modes (Pipe vs File vs FastFile), the high-performance file system FSx for Lustre, and the data sharding concept for distributed training.
SageMaker training jobs have multiple modes for bringing data from S3 to the container.