Day 4 - Data Storage Strategy: Partitioning, Format Optimization, Training Preparation

📌 Key Takeaways

• Storage is not a question of "where you put it" but of "whether you lay it out in advance so the reading side can read quickly and cheaply."
• Make the partition key the column you filter on most often (usually the date). Splitting too finely creates the small files problem.
• The default format is Parquet + Snappy — the triple of columnar, compression ratio, and splittable.
• CSV/JSON compressed with Gzip is non-splittable, which blocks parallel reads. "Is it splittable?" matters more than "is it compressed?"
• SageMaker input modes are File / Pipe / FastFile. When the data is bigger than the disk, use Pipe/FastFile so the GPU is not starved.

Partitioning: Don't Touch Data You Won't Read

Even for the same data, how you store it can change query cost by dozens of times and training speed by several times.

Partitioning means splitting data into folders by a specific key (usually the date). If a query reads only the partitions it needs through a WHERE condition, the rest is not scanned — this is partition pruning.

s3://ml-datalake/events/
 ├─ year=2026/month=05/day=01/part-0000.parquet
 ├─ year=2026/month=06/day=24/part-0000.parquet
 └─ year=2026/month=06/day=25/part-0000.parquet   ← with WHERE day='25', only this is scanned

Folder names in the key=value form such as year= and month= are called Hive-style partitioning. Glue Crawler and Athena recognize this convention automatically and register them as partition columns.

-- Filtering by the partition key scans only that partition's Parquet files
SELECT customer_id, COUNT(*) AS cnt
FROM ml_datalake.events
WHERE year = '2026' AND month = '06' AND day = '25'
GROUP BY customer_id;

Criteria for choosing a partition key — choose wrong and the effect is zero.

The effect of partitioning appears "only when the partition key is used in the query condition." If you partition by date but the query filters only with WHERE customer_id = ..., it fully scans every date partition. At the same time, if you split too finely (e.g., by the second), small files grow into the millions, causing metadata overhead and the "small files problem." A medium-cardinality column such as date (year/month/day) or region is usually the right line.

File Formats: Row-Oriented vs Columnar

In ML data storage, the choice of format ties directly to cost and speed. The key is the difference between row-oriented and columnar.

There are three reasons Parquet is the de facto standard for analytics and ML.

1. Being columnar, it reads only the needed columns, so I/O is low.
2. The compression ratio is high per column (because values of the same type are gathered).
3. It carries the schema and statistics (min/max) in the file so it can skip unnecessary blocks.

import pandas as pd
 
df = pd.read_csv("s3://raw/events.csv")
 
# Convert to Parquet + Snappy compression (Athena/SageMaker read it quickly)
df.to_parquet(
    "s3://ml-datalake/curated/events.parquet",
    engine="pyarrow",
    compression="snappy",
)

Choosing a compression codec

Snappy has a medium compression ratio but compresses and decompresses fast, so it suits data read often such as queries and training (the Parquet default). Gzip has a high compression ratio that cuts storage cost but uses more CPU — it fits archives that are rarely read. ML training data is read repeatedly, so the Snappy + Parquet combination is usually the answer.

⚠️ Trap: It is not "compression is always good." If you compress CSV/JSON with Gzip, the file becomes non-splittable, so several workers cannot divide up and read a single file in parallel. Parquet is splittable in units of internal blocks (row groups), so it preserves parallelism in distributed training and queries. When handling big data, "is it a splittable format?" matters more than "is it compressed?"

[One Gzip CSV file]        Worker1 ████████████████  Worker2 (idle)
                            → One worker has to read everything sequentially

[Parquet, several shards]  Worker1 ████  Worker2 ████  Worker3 ████  Worker4 ████
                            → Split by row group/file and read at the same time

SageMaker Training Data Preparation

The destination of a storage strategy is "how fast a SageMaker training job can receive data from S3." SageMaker provides input modes for delivering S3 data to the training container.

[File]      ████ full copy (tens of minutes) ████ ▶ training starts ──────────▶
[Pipe]      ▶ training starts ─── keeps being fed by streaming ───────────▶
[FastFile]  ▶ training starts ─── loads only the needed files on demand ──────▶

from sagemaker.inputs import TrainingInput
 
# Stream large data in with Pipe mode
train_input = TrainingInput(
    s3_data="s3://ml-datalake/curated/train/",
    input_mode="Pipe",            # File / Pipe / FastFile
    content_type="application/x-parquet",
)
estimator.fit({"train": train_input})

When you place data in S3, it is also important to split it into several files (shards) of moderate size. A single giant file is hard for multiple instances to divide and read in distributed training. SageMaker can hand files out to instances with the ShardedByS3Key distribution type.

⚠️ Trap: Turning on ShardedByS3Key is not the end of it. Sharding works at the S3 object level, so applying it to a single 100GB file distributes nothing. The data has to be divided into several objects for it to take effect.

In ML training, data input often becomes a bigger bottleneck than the GPU. If the GPU computes a batch in 0.1 seconds but reading the data takes 0.5 seconds, the GPU sits idle most of the time (GPU starvation). Pipe/FastFile mode, an efficient format like Parquet, and appropriate sharding are all devices to "avoid starving the expensive GPU." That is why a storage strategy is a training cost strategy.

One team kept several TB of click logs in S3 as a single giant CSV and trained in File mode, wasting 40 minutes on data copying for every experiment. When they re-stored the data as Parquet shards partitioned by date and switched to Pipe mode, the copy wait disappeared and doing EDA beforehand with Athena also became easier. A single storage format governed the speed of experiment iteration.

Storage Design Checklist

Seeing the Difference Storage Design Makes in Numbers

Even for the same 5TB dataset and the same query, the amount read differs like this depending on the layout.

• If you use only 5 of 100 columns and query only one month, the two levers multiply and scan volume drops dramatically.
• This saving is reflected not only in Athena charges but also directly in SageMaker training's data loading time.

Balancing File Size and Count

Input Mode Selection Flow

Does the training data fit on the instance disk?
   │
   ├─ Yes ─── Is a full shuffle/random access every epoch important?
   │           ├─ Yes → File mode
   │           └─ No → File or FastFile
   │
   └─ No ─── Is access sequential?
               ├─ Yes → Pipe mode (no download wait)
               └─ No (random access to only some files) → FastFile mode

Checking Whether the GPU Is Idle and Responding

When GPU utilization is low, upgrading the instance type to a bigger GPU is mostly a wrong answer. If the bottleneck is input rather than computation, a more expensive GPU just sits idle longer. Look first at the storage format, input mode, and sharding.

One-Line Discriminators for Storage Strategy

In the next note we wrap up Week 2 with a comprehensive review of this week's data ingestion and storage.

📖 Glossary

• Partitioning: Splitting data into folders by a key such as the date. It lets you avoid touching data you will not read.
• Partition pruning: An optimization that reads only the partitions matching the condition and skips the rest entirely.
• Small files problem: A phenomenon where files are split too finely and per-file overhead dominates processing time.
• Row-oriented: A method that stores the values of a row side by side. Advantageous for work that often uses an entire row.
• Columnar: A method that gathers and stores the values of the same column. Advantageous for analytics and training that read only some columns.
• Row group: The block unit into which Parquet divides data internally. Thanks to this, split reads are possible even when compressed.
• Non-splittable: A state where a file cannot be read from the middle, so one worker must process it whole.
• Snappy: A compression codec that chooses speed over compression ratio. The default choice for training data read often.
• Input mode: How S3 training data is put into the container. Three kinds: File, Pipe, FastFile.
• Pipe mode: A method that streams from S3 and starts training right away without waiting for a full download.