Day 1 - Integrated Review of Domains 1 & 2: Ingestion, Transformation & Storage Management

📌 Key Takeaways

• Domain 1 (ingestion and transformation, 34%) plus Domain 2 (store management, 26%) make up 60% of the scored questions. Do not study the services one by one; tie them into a single thread: "data comes in, gets transformed, and gets stored."
• The source decides the ingestion service: direct consumption or replay = Kinesis Data Streams, unmanaged-free loading = Data Firehose, Kafka assets = MSK, database changes = DMS (CDC), files = S3 + Glue.
• Transformation is chosen by scale and operational burden: serverless ETL = Glue; large scale, fine-grained tuning, or existing Spark/Hive = EMR. Incremental ETL means Glue job bookmarks.
• The store is decided by how the data is read: lake = S3, analytic DW = Redshift, serverless SQL = Athena, key-value = DynamoDB, OLTP = RDS/Aurora.
• In Redshift, DISTKEY reduces shuffling and SORTKEY reduces scanning, and Spectrum keeps cold data in S3. The shared metastore is the Glue Data Catalog, and central fine-grained permissions come from Lake Formation.

Tying the Two Domains into One Thread

This is the final week. Today we review the two areas with the largest exam weight as a single flow. Rather than memorizing individual services, the key to passing is to see them as a pipeline: "data arrives, is loaded, is transformed, and is stored." A typical exam scenario leaves one or two of the five slots below blank and asks you to fill them in.

1. Ingestion: Kinesis/Firehose (streaming) or DMS (CDC) → S3 raw zone
2. Catalog: a Glue crawler infers the schema → registered in the Data Catalog
3. Transformation: a Glue job (incremental with bookmarks) or EMR → S3 curated zone (Parquet)
4. Storage/query: ad hoc queries with Athena, or load into Redshift for BI
5. Governance: fine-grained per-team permissions with Lake Formation

The exam guide splits the two domains into eight tasks. If you know which week of this track touches which task, you can pick your weak spots and go back to them.

Domain 1: Ingestion Pattern Summary

Ingestion splits broadly into streaming and batch. When the exam shows keywords like "real time" or "second-level latency," it is asking about streaming; when it shows "periodic," "bulk," or "hourly," it is asking about batch.

💡 Related theory: Choose Kinesis Data Streams when you need "direct consumption, reprocessing, and ordering," and Firehose when the goal is "managed-free loading." Firehose has no shard or consumer concept; it delivers automatically to the destination based on buffer size and interval.

The moment you pick an ingestion service, you also fix its replay range. That is why the exam guide lists "replayability of ingestion pipelines" as its own item.

• When a reprocessing requirement exceeds the retention period, the answer is not the stream but the S3 original + batch reprocessing. That is why the combination "real-time analytics + original retention" (KDS → real-time consumers, with Firehose loading the same stream into S3) shows up repeatedly as the correct answer.

Domain 1: Transformation Summary

Choosing a transformation engine comes down to "serverless or not + data scale + coding style."

• AWS Glue: Serverless Spark ETL. The default choice for small to medium scale, schedule- and trigger-based ETL. Its key feature is the Glue job bookmark, which skips data that has already been processed to enable incremental ETL.
• Amazon EMR: A managed Hadoop/Spark cluster. For large scale, fine-grained tuning, or reusing existing Spark/Hive code. Cost optimization is Spot Instances + instance fleets.
• Glue Studio / DataBrew: Visual/no-code transformation. DataBrew is for analysts to clean and profile data.

# Glue bookmark: process only new data since the last run (incremental ETL)
job.init(args['JOB_NAME'], args)
dyf = glueContext.create_dynamic_frame.from_catalog(
    database="raw", table_name="events",
    transformation_ctx="dyf"  # the bookmark tracks state by this ctx
)
# ... transformation ...
job.commit()  # the bookmark state is saved at commit time

💡 Related theory: A Glue bookmark works only when transformation_ctx and job.commit() are both present. For problems like "incremental ETL," "prevent reprocessing of data already processed," or "a scheduled job reads everything again each time," enabling the bookmark is the correct answer.

Before choosing the bookmark as the answer, there are three conditions to check (Week 4 Day 2).

• The default value of the bookmark option is Disable. Even if the code has transformation_ctx and job.commit(), the job reads everything every time when the option is off.
• An S3 source identifies new data by the object's last modified time, and a JDBC source by the bookmark key. So it cannot catch UPDATEs and DELETEs of existing rows in a JDBC source. If the requirement is "reflect changes too," the answer is DMS CDC.
• If the job fails before job.commit(), the state is not updated and the next run reads the same input again. A bookmark is also at-least-once in nature, so design the output to be idempotent, such as overwrite or MERGE.

Task 1.4: What Comes Out of Programming Concepts

This is a task for which this track has no dedicated week. It asks not about syntax but about "which tool does what." The exam guide explicitly places knowledge of the syntax of any specific language out of scope.

• If the downstream system (such as RDS) can handle only a small number of connections, cap Lambda with reserved concurrency. If concurrency is left unlimited, database connections are exhausted as Lambda scales out.

Domain 2: Choosing a Data Store

Map stores by "structure, access pattern, and query style."

Domain 2: Redshift Operations Essentials

On the exam, Redshift is about distribution keys and sort keys.

• Distribution style (DISTSTYLE): KEY (places rows with the same join column value on the same node, reducing shuffling), ALL (replicates a small dimension table in full), EVEN (spreads evenly), AUTO (Redshift chooses automatically).
• Sort key (SORTKEY): Speeds up range filters and time-range queries. Blocks are skipped using zone maps.
• Redshift Spectrum: Queries external data in S3 from Redshift without loading it. A tier separation in which hot data stays in Redshift and cold data stays in S3.

CREATE TABLE fact_sales (
  sale_id bigint, customer_id bigint, sale_ts timestamp, amount decimal(10,2)
)
DISTSTYLE KEY DISTKEY (customer_id)   -- optimizes the join with the customer dimension
SORTKEY (sale_ts);                    -- speeds up time-range queries

When joining a large fact table with a small dimension table, use DISTSTYLE ALL if the dimension is small; when joining two large tables, put a DISTKEY on the common join key on both sides so that no data moves between nodes (shuffle).

Let us go one level deeper into how the four distribution styles and two sort key types actually work.

• What a zone map really is: Redshift stores column data in 1 MB blocks and keeps each block's minimum and maximum values as metadata. When a range condition arrives, it looks only at those values and skips whole blocks. If the data is not sorted, values are scattered across many blocks and there are no blocks to skip.
• AWS recommends creating tables with SORTKEY AUTO and DISTSTYLE AUTO and leaving them to automatic table optimization. However, when the exam stem points at the cause, as in "reduce data movement between nodes," the choice that picks the distribution key yourself is the answer.
• The cue for ALL is not just "small" but "small and rarely changing." A choice that applies ALL to a large table or a frequently updated table is wrong.

Domain 2: Lake Formation and the Catalog

This is permission and metadata management for the data lake.

• Glue Data Catalog: The central metastore shared by Athena, Redshift Spectrum, and EMR. Crawlers infer schemas automatically.
• Lake Formation: Central, fine-grained permissions for the data lake. Table, column, row, and cell level permissions and LF-Tag based policies. Instead of writing IAM policies for each S3 object one by one, you manage data permissions centrally.

For the scenario "several teams share an S3 data lake, and we want to control access at the column/row level and manage it in one place," the answer is almost always Lake Formation. IAM bucket policies alone cannot provide column- or row-level control.

Paths Into and Out of Redshift

A stem that says "load data into Redshift" has different answers depending on the source and freshness.

• Why COPY of one large file is slow: a file that cannot be split is read serially by a single slice. It cannot use Redshift's parallelism.
• The complete form of "hot in Redshift, cold in S3" is to UNLOAD old data as Parquet and attach it back with a Spectrum external table.

Data Lifecycle: Different Expiry Mechanisms for Each Store

For S3 storage classes, the minimum storage duration is the trap. If you delete or move an object before it, you are charged for the remaining period.

• Lifecycle transitions flow only from top to bottom (the waterfall model). From Glacier Deep Archive you cannot move back to another class with a rule; you have to copy the restored copy.
• Objects smaller than 128 KB are excluded from Lifecycle transitions by default, because the transition request fee can exceed the savings (this default changed in September 2024, so older material may say "they are transitioned to Glacier").
• Athena cannot read objects stored in Glacier Flexible Retrieval or Deep Archive without restoring them. If the condition is "we query it with SQL occasionally," a choice that moves the curated zone to an archive class is eliminated.

New Store Topics in the Exam Guide

Version 1.1 of the exam guide, published in December 2025, added vectors and open table formats to Domain 2. Prepare at the level of distinguishing terms rather than depth.

Symptom → Cause → Action

Scenario → Answer Decision Table

These are situations that do not overlap with the practice questions, with pairs where changing one condition changes the answer.

Tomorrow we combine Domains 3 and 4, on running the pipelines built this way reliably and protecting them safely.

📖 Glossary

• Replayability: The ability to send data that has already passed through again for reprocessing. The retention period sets its range.
• Bookmark key: The reference column Glue bookmarks use to remember "how far we have read" in a JDBC source.
• Slice: The unit of parallel processing into which a Redshift compute node is divided. The distribution style decides which slice a row goes to.
• Shuffle (redistribution): Moving rows between nodes for a join or aggregation. It does not occur when the distribution key matches.
• Zone map: The minimum and maximum values stored for each block. It lets the engine skip blocks that do not match the condition without reading them.
• Materialized view: A view whose query result is precomputed and stored. It keeps up with the latest state through refresh.
• Federated query: A query that reads tables in another database directly, without moving the data.
• Minimum storage duration: The minimum number of days a storage class requires. Even if you delete or move the object earlier, you are billed for the remaining period.
• TTL: A DynamoDB feature that automatically deletes an item once the expiration timestamp written on it has passed.
• Embedding: Text or an image converted into a numeric vector that reflects its meaning. A vector index uses it as its search target.