Day 5 - Week 2 Comprehensive Review — Data Collection & Storage Recap

📌 Key Takeaways

• The big picture of this week is a data lake centered on S3. Every service shares S3 as its input and output.
• Pick the ingestion tool by the shape in which data arrives — Kinesis for streaming, DMS for databases, DataSync for files.
• Storage is processed for training — Parquet + Snappy + date partitioning + sharding.
• Glue is the hub for cataloging and transformation — Crawlers register schemas, and ETL Jobs/DataBrew transform the data.
• Athena bills by bytes scanned, so format and partition design is the cost. The SELECT * habit inflates the bill.

The Entire Data Pipeline at a Glance

Rather than memorizing this week's services one by one, it is more efficient for exam prep to place them on the flow of "raw data comes in and becomes a trainable dataset."

[Source]
  │  ① Ingestion
  ├─ Streaming ──> Kinesis Data Streams / Firehose ─┐
  └─ Batch ──────> DMS / DataSync / direct upload ──┤
                                                    v
                                           ┌─────────────────────┐
                                           │   S3 data lake      │  ② Storage
                                           │ (single source of   │
                                           │  truth)             │
                                           └─────────────────────┘
                                                    │  ③ Catalog & transform
                                           Glue Crawler → Data Catalog
                                           Glue ETL Job (cleaning, format conversion)
                                                    │  ④ Query & explore
                                           Athena (serverless SQL, EDA)
                                                    │
                                                    v
                                           SageMaker training (Parquet, Pipe mode)

The key point is that S3 is at the center. Every service uses S3 as its input and output, and training ultimately reads its data from S3 as well.

This structure is the classic "data lake." A data warehouse loads structured data into a predefined schema up front (schema-on-write), whereas a data lake first piles raw data into S3 as it is and applies the schema when the data is read (schema-on-read). ML works with structured, semi-structured, and unstructured data, and it does not know in advance which features will turn out to be useful, so the data lake model of "collect everything first and interpret it later" fits well.

Ingestion: Streaming vs Batch

The tool depends on the shape in which the data arrives. This is an easy distinction to mix up.

You need to tell the two Kinesis siblings apart. Data Streams lets you attach your own consumers to process data in real time and requires you to manage shards (flexible, low latency). Firehose is fully managed: it buffers the data and then delivers it automatically to S3/Redshift (when you only want delivery without writing code).

import boto3
kinesis = boto3.client("kinesis")
 
# Put a real-time event into Kinesis Data Streams
kinesis.put_record(
    StreamName="clickstream",
    Data=b'{"user_id": "u123", "event": "click", "ts": "2026-06-25T10:00:00Z"}',
    PartitionKey="u123",     # Same key → same shard → order preserved
)

If the need is "real-time, but just pile it into S3 without code," it is Firehose; if "multiple apps each process the real-time stream and replay is also needed," it is Data Streams. The PartitionKey sends records with the same key to the same shard, which preserves order — use user_id as the key when the per-user event order matters.

⚠️ Trap: Reducing the stream to 1 shard to preserve order is the wrong answer. Order is guaranteed within a shard, so with a well-chosen PartitionKey you can keep several shards and still preserve order per key. Cutting down to 1 shard only reduces overall throughput.

Storage: Format and Partitioning

Collected data is not left as it is; it is processed to favor training. Summarizing the gist of the storage strategy from yesterday:

These five decisions determine both the Athena query cost (billed by bytes scanned) and the SageMaker training speed.

Common storage mistakes and their symptoms

Catalog & Transform: Glue

The center of the ETL that turns raw data into a trainable form is AWS Glue.

# Glue PySpark ETL: clean the CSV and save it as partitioned Parquet
from awsglue.context import GlueContext
from pyspark.context import SparkContext
 
glueContext = GlueContext(SparkContext.getOrCreate())
df = glueContext.create_dynamic_frame.from_catalog(
    database="ml_datalake", table_name="raw_events"
).toDF()
 
df.write.partitionBy("year", "month").mode("overwrite") \
    .parquet("s3://ml-datalake/curated/events/")

The Glue Data Catalog acts as the "single source of truth for metadata." Once a Crawler registers a schema, Athena, Redshift Spectrum, and EMR share the same table definition. Separating the data (S3) from the metadata (Catalog), so that several query engines can each read the same data in their own way, is the core design of a data lake architecture.

Query & Explore: Athena

This is the stage where you read and understand the prepared data. Athena queries S3 directly with serverless SQL and bills by the amount of data scanned ($5/TB). That is why Parquet + partitioning + compression translate directly into cost savings. It is the main venue for EDA (checking distributions, missing values, class imbalance, and data leakage), and with CTAS you can turn query results straight into a Parquet training dataset.

-- Reduce the scan volume with a partition filter while checking class imbalance
SELECT churned, COUNT(*) AS cnt
FROM ml_datalake.events
WHERE year = '2026' AND month = '06'
GROUP BY churned;

⚠️ Trap: If you forget that Athena cost is proportional to the amount scanned and full-scan everything with SELECT *, it gets expensive. The habit of selecting only the columns you need (taking advantage of Parquet's column skipping) and filtering by partition keys cuts cost by tens of times.

Week 2 One-Line Checklist

Service Wrong-Answer Matching Summary

On the exam, wrong answer options usually slip in "a service from a different stage." If you pin down which stage each service belongs to, you can filter them out.

This Week Revisited from a Cost Perspective

A single storage design decision drives the cost of every later stage. Parquet + date partitioning + appropriate sharding pays off in three stages at once: catalog, query, and training.

Five Things That Ruin a Pipeline

This week's services are asked about not as "what does it do" but as "under what conditions is it the best choice." Memorizing service names and features is not enough; you must be able to compare them along four axes: latency, management overhead, cost, and data size.

Week 2 in One Sentence

"Pile up the originals in S3 as immutable data, catalog and transform them with Glue and reorganize them into Parquet + date partitioning, then explore them with Athena and let SageMaker read them as they are for training."

Next week (Week 3) covers feature engineering, which turns the data prepared this way into a form the model can read, along with checking the data for bias and quality.

📖 Glossary

• Data pipeline : The series of steps that move and refine data from the source to the training dataset.
• Ingestion : The process of bringing external data into AWS (mainly S3). It is divided into streaming and batch.
• PartitionKey : The value in Kinesis that decides which shard a record goes to. The same key goes to the same shard, so order is preserved.
• Replay : Re-reading stream data within the retention period from the beginning. Possible only with Data Streams.
• Buffering : The way Firehose gathers data up to a certain size or time and then flushes it all at once.
• schema-on-write : The traditional DB approach that enforces the schema at the moment data is inserted. Contrasts with a data lake.
• Glue Crawler : A tool that scans a data store, automatically discovers the schema and partitions, and registers them in the catalog.
• Data Catalog : A central store of table definitions shared by several query engines. It does not hold the data itself.
• Sharding : Splitting data into several appropriately sized files so that parallel reads become possible.
• Scan-based billing : A model where the charge is proportional to the bytes a query actually reads. The key constraint in Athena design.