Day 5 - Week 1 Comprehensive Review

Over the past four days we laid the foundation of data engineering. Day 1 covered the role and the pipeline, OLTP/OLAP, and lake/warehouse; Day 2 split batch from streaming; Day 3 unfolded the map of AWS data services; and Day 4 dealt with data formats and schema evolution. They may look like scattered pieces, but in fact they thread together into a single pipeline.

Today, instead of re-listing individual concepts, we follow one scenario from beginning to end to see how the four days of knowledge interlock. The exam doesn't test fragmented memorization — it asks "what would you choose in this situation" — so reviewing through these connections is the most effective approach.

Week 1 on One Page

[Day1] Roles & concepts   [Day2] Processing  [Day3] Services      [Day4] Formats
 Data engineer             Batch              Ingest Kinesis/DMS   CSV/JSON (raw)
 = pipeline owner          vs                 Store S3/Redshift    Parquet/ORC (analytics)
 OLTP vs OLAP              Streaming          Process Glue/EMR     Avro (streaming)
 Lake vs Warehouse         (latency/          Analyze Athena/QS    Schema evolution
                            throughput)       Governance LakeFmt

The keywords of the four days are ultimately different facets of one question: "how do you ingest, store, process, analyze, and govern data."

💡 Related theory: The single principle running through all these judgments is "the workload determines the design." OLTP or OLAP, batch or stream, row-based or columnar — the answers all come from the workload characteristic of "how is the data read and written." Data engineering's way of thinking is not to pick the technology first and force the workload to fit, but to analyze the workload and then choose the technology.

Threading It Together with a Scenario

Let's follow the requirements of an e-commerce company. Each requirement is tagged with which Week 1 concept it touches.

Requirement 1. "Make the order data analyzable without putting load on the production order DB."

• The production DB is OLTP (Day 1) → heavy aggregation must not run directly on it
• Replicate only the changes to the analytics environment with DMS CDC (Day 3)
• The analytics environment is OLAP (Day 1) → Redshift or S3+Athena

Requirement 2. "I want to catch fraudulent transactions the moment a payment happens."

• Immediacy required → streaming (Day 2)
• Ingest payment events with Kinesis Data Streams (Day 3)
• The schema may change, so Avro + Schema Registry (Day 4)

Requirement 3. "Yesterday's revenue report, automatically, every morning."

• Periodic, bounded data → batch (Day 2)
• Process a day's worth with Glue (Day 3), load as Parquet (Day 4)
• Query and visualize with Athena/QuickSight (Day 3)

-- The analytics query for Requirement 3 — fast and cheap because it's stored as Parquet
SELECT region, SUM(amount) AS revenue
FROM orders          -- Parquet in S3, registered in the Glue Catalog
WHERE order_date = DATE '2026-06-25'   -- partition pruning
GROUP BY region;

A single company can have all three requirements at once, each using different paradigms, services, and formats. The key insight is that there is no "one right answer" — there is a fitting combination for each requirement.

💡 Related theory: Merge Requirements 1–3 into one picture and you naturally get a lakehouse. All raw data (order CDC, payment streams, logs) gathers in the S3 data lake, gets refined into Parquet through batch and streaming processing, and Athena, Redshift, and QuickSight analyze on top of it. This is why Day 1's "lake vs warehouse" gets "unified into the lakehouse" in the modern era.

Sorting Out the Commonly Confused Pairs

These are the pairs that show up as exam traps.

Getting the distinction criteria for just these eight pairs right lets you solve a good share of the scenario questions in Week 1's scope.

💡 Related theory: Nearly every pair in this table shares the same tension: "flexibility and generality vs efficiency and specialization." The lake is flexible but needs management; the warehouse is efficient but rigid. Serverless (Glue/Athena) is convenient but weaker on fine-grained control; clusters (EMR/Redshift) are powerful but carry operational burden. Once you are conscious of the trade-off, memorization turns into understanding.

The Exam Perspective in One Line

DEA-C01 ultimately asks about these four domains (reconfirming Day 1).

1. Ingestion & Transformation (34%)  ← Day2 batch/stream + Day3 Kinesis/Glue/EMR
2. Store Management (26%)            ← Day1 lake/warehouse + Day3 S3/Redshift + Day4 formats
3. Operations & Support (22%)        ← (Week 2 onward) orchestration, monitoring
4. Security & Governance (18%)       ← Day3 LakeFormation/IAM/KMS

Week 1 laid nearly all the groundwork for domains 1 and 2. From next week we go into the deep details of each service (partitioning strategies, Glue job tuning, Kinesis shards, and so on), but remember that all of it rests on the big picture we drew today.

Wrapping Up

Week 1's conclusion is simple. The data engineer is the person who builds pipelines, and that pipeline flows through ingestion → storage → processing → analytics, wrapped by governance. Every design judgment (OLTP/OLAP, batch/stream, lake/warehouse, row-based/columnar) ultimately comes down to "what does the workload demand."

If you have this way of thinking in hand, Week 1 is a success. From Week 2 we put flesh on this skeleton and get into actually working with each service.