Day 5 - Week 1 Comprehensive Review

📌 Key Takeaways

• The conclusion of Week 1: a data engineer builds the ingest → store → process → analyze pipeline, and governance wraps the entire span.
• Every choice (OLTP/OLAP, batch/stream, lake/warehouse, row-based/columnar) comes from "what does the workload demand?"
• Break a passage down with four questions: what is the source → how fresh must it be → who reads it and how → what are the operational-overhead and cost conditions.
• Wrong options are ones that are "possible functionally but violate one condition." Qualifiers such as least operational overhead, lowest cost, and fewest code changes decide the answer.
• If you can state the distinguishing axis of the 8 confusing pairs (Streams/Firehose, Glue/EMR, Athena/Redshift, IAM/Lake Formation, and so on) in one word each, you pass.

Week 1 on One Page

Over the past four days we laid the foundation of data engineering. On Day 1 we saw roles and pipelines, OLTP/OLAP, and lake/warehouse; on Day 2 we separated batch from streaming; on Day 3 we spread out the map of AWS data services; and on Day 4 we covered data formats and schema evolution. They look like scattered pieces, but in fact they are threaded together into a single pipeline.

Today, instead of listing the individual concepts again, we follow one scenario from start to finish to see how the knowledge of the four days interlocks. Because the exam asks not for fragmentary memorization but "what would you choose in this situation," looking at it connected like this is the most effective review.

[Day1] Roles/concepts     [Day2] Processing   [Day3] Services       [Day4] Formats
 Data engineer             Batch               Ingest Kinesis/DMS    CSV/JSON (raw)
 = owns the pipeline       vs                  Store S3/Redshift     Parquet/ORC (analytics)
 OLTP vs OLAP              Streaming           Process Glue/EMR      Avro (streaming)
 Lake vs warehouse         (latency/throughput) Analyze Athena/QS    Schema evolution
                                               Governance LakeFmt

The keywords of the four days are, in the end, different aspects of a single question: "how do we ingest, store, process, analyze, and control data?"

💡 Related theory: The single principle running through all of these judgments is "the workload determines the design." Whether OLTP or OLAP, batch or stream, row-based or columnar — the answer in every case comes from the workload characteristic of "how is the data read and written?" The way of thinking in data engineering is not to pick a technology first and force the workload to fit, but to analyze the workload and then pick the technology.

Threading It Together with a Scenario

We follow the needs of an e-commerce company. We have marked which Week 1 concept each need touches.

Need 1. "Don't put load on the production order DB, and make that data analyzable."

• The production DB is OLTP (Day 1) → heavy aggregations must not be run on it directly
• Replicate only the changes to the analytics environment with DMS CDC (Day 3)
• The analytics environment is OLAP (Day 1) → Redshift or S3 + Athena

Need 2. "I want to catch fraudulent transactions at the moment of payment."

• Immediacy is needed → streaming (Day 2)
• Ingest payment events with Kinesis Data Streams (Day 3)
• The schema may change, so Avro + Schema Registry (Day 4)

Need 3. "Automatically produce yesterday's sales report every morning."

• Periodic, bounded data → batch (Day 2)
• Process a day's worth with Glue (Day 3) and load it as Parquet (Day 4)
• Query and visualize with Athena/QuickSight (Day 3)

-- The analytic query for Need 3 — stored as Parquet, so it's fast and cheap
SELECT region, SUM(amount) AS revenue
FROM orders          -- Parquet in S3, registered in the Glue Catalog
WHERE order_date = DATE '2026-06-25'   -- partition pruning
GROUP BY region;

A single company can have all three needs at once, and each uses a different paradigm, service, and format. The key is that there is not "one right answer" but a combination that fits each need.

💡 Related theory: Merging Needs 1–3 into one picture naturally gives you a lakehouse. All raw data (order CDC, the payment stream, logs) gathers in the S3 data lake, goes through batch and streaming processing and is refined into Parquet, and Athena, Redshift, and QuickSight analyze on top of it. This is why Day 1's "lake vs. warehouse" is "unified into a lakehouse" in modern practice.

Summary of Frequently Confused Distinctions

These are the pairs that appear as traps on the exam.

Just getting the distinguishing criterion of these 8 pairs right can solve a good share of the scenario questions in the Week 1 scope.

Almost every pair in this table shares the same tension structure: "flexibility and generality vs. efficiency and specialization." A lake is flexible but needs management, and a warehouse is efficient but rigid. Serverless (Glue/Athena) is convenient but weak on fine-grained control, and clusters (EMR/Redshift) are powerful but carry operational overhead. Being conscious of the trade-offs turns memorization into understanding.

One-Line Summary from the Exam's Perspective

DEA-C01 ultimately asks about these 4 domains (reconfirming Day 1).

1. Ingestion and transformation (34%)   ← Day 2 batch/stream + Day 3 Kinesis/Glue/EMR
2. Storage and management (26%)         ← Day 1 lake/warehouse + Day 3 S3/Redshift + Day 4 formats
3. Operations and support (22%)         ← (Week 2 onward) orchestration and monitoring
4. Security and governance (18%)        ← Day 3 LakeFormation/IAM/KMS

Week 1 has laid almost all of the foundation for domains 1 and 2. From next week we go into the deep details of each service (partitioning strategy, Glue job tuning, Kinesis shards, and so on), but all of it rests on the big picture we drew today.

The Order for Solving a Passage: Four Questions

If you memorize the Week 1 concepts separately they scatter; if you bundle them in question order they become a decision procedure. When you meet a scenario passage, underline in the following order.

① What is the source?          DB / files / events and logs / SaaS
        ↓                      → decides the ingestion service (Day 3)
② How fresh must it be?        within seconds / minutes / hours / a day
        ↓                      → decides batch, streaming, or near-real-time (Day 2)
③ Who reads it, and how?       single-row lookup / large aggregation / search / dashboard
        ↓                      → decides the store and format (Day 1, Day 4)
④ What conditions are attached?  least operational overhead / lowest cost / fewest code changes / fine-grained permissions
                               → narrows to one among the candidates at the same stage

• ④ most often decides the answer. Through ① to ③ two or three options survive, and a single last qualifier settles it.

Five Molds That Wrong Options Are Made From

Question writers build options that are not "wrong technology" but "the wrong choice for this situation." Knowing the molds speeds up elimination.

• Especially the fifth mold: a serverless service has no knob for increasing size. Athena has no instances, so performance is improved on the data side (format, compression, partitions, file size).

Scenario Practice: From Clue to Answer

Below are new situations that do not overlap with the practice questions. Cover the clue column, state the answer, and then check.

Week 1 Numbers and Defaults on One Page

The fewer numbers to memorize, the better. We have picked out only those that are actually used to discriminate passages within the Week 1 scope.

• Limit and pricing figures change. The exam asks not for exact numbers but for how it behaves, such as "there is a default and it is adjustable" or "it acts on the condition reached first."

Failures Arising in One Pipeline and Their Causes

If you flip the concepts into "what goes wrong and what symptom appears," all of Week 1 connects again.

Most of the remedies in the table above are a matter of "how you lay out the data." Scaling up the cluster is the last resort, and the data engineer's job is to first fix format, partitions, file size, and separation (production vs. analytics, originals vs. processed copies). This is also why DEA-C01 asks about design judgment more than service names.

If you now hold this way of thinking in your hands, Week 1 is a success. From Week 2, we flesh out this skeleton and move on to how to actually work with each service.

📖 Glossary

• Workload : The nature of the work a system must handle. It describes how data is read and written.
• OLTP / OLAP : Transaction processing that reads and writes a few rows quickly, versus analytical processing that gathers and aggregates many rows.
• Lakehouse : A structure that layers a table format and a catalog on an S3 data lake so it can be queried like a warehouse.
• schema-on-read / schema-on-write : Applying structure at read time, versus enforcing structure at store time.
• Qualifier : A condition phrase in a passage such as "least operational overhead" or "most cost-effective." It decides one answer among the candidates.
• Process of elimination : A solving method of erasing options that violate a condition one by one to leave the answer.
• Partition pruning : Reading only the partition folders that match the WHERE condition and skipping the rest.
• Small-files problem : The phenomenon where too many very small files make listing and opening costs grow and queries slow down.