Day 1 - What Is Data Engineering

ETL (Extract → Transform → Load)
  Extract → Transform → Load
  Pull from the source → transform in the middle → put into the destination

ELT (Extract → Load → Transform)
  Extract → Load → Transform
  Pull from the source → load everything first → transform inside the destination

Traditional ETL finished transformations in advance on a separate server, then loaded only clean data into the warehouse — because warehouse storage and compute were expensive. But with the cloud came cheap storage like S3 and powerful query engines like Redshift and Athena, and the flow changed. Load all the raw data first (L), then transform it inside the destination when needed (T) — ELT became commonplace.

A typical pipeline on AWS looks like this.

[Source DB / logs / streams]
        │ Extract
        ▼
   [S3 raw zone]  ← store everything cheaply first
        │ Transform (Glue / EMR / Athena)
        ▼
  [S3 curated zone / Redshift]  ← analyzable form
        │
        ▼
   [QuickSight / SageMaker]  ← consumption

💡 Related theory: The choice between ETL and ELT is a question of "where do you transform." Because ELT preserves the raw data, you can reprocess it later when transformation logic changes without going back to the source (reprocessing). This principle of "never throw away the raw data" is the core philosophy of the data lake, which we'll see next.

OLTP vs OLAP: Two Worlds of Processing

This is the most important contrast for understanding data systems. Even though both are called "databases," their purposes are polar opposites.

OLTP handles a small number of rows accurately and quickly, like "look up customer 1234's balance and deduct 10,000 won." OLAP sweeps through hundreds of millions of rows and summarizes, like "aggregate the last three years of regional revenue by month."

One of the data engineer's common missions is to safely replicate and move data from the OLTP system (the operational DB) into the OLAP system (the analytics environment) without putting load on the OLTP system. Firing heavy aggregation queries directly at the operational DB slows down the service.

💡 Related theory: The reason OLAP uses columnar storage comes from this workload difference. Aggregation queries usually read only a few of all the columns, so storing data column by column lets you scan only the columns you need, dramatically reducing I/O. This is the background for why Redshift and Parquet are columnar. Columnar vs row-oriented storage is covered in depth on Day 4.

Data Lake vs Data Warehouse

These are the two pillars of storage design. They are not competitors — they are usually used together.

• Data Warehouse: Define the schema first (schema-on-write) and load only cleansed, structured data. The strict structure makes queries fast and consistent, but unstructured data or data whose schema changes frequently is hard to handle. On AWS, this is Redshift.
• Data Lake: Store structured, semi-structured, and unstructured data alike in its raw form (schema-on-read), applying structure at read time. Flexible and cheap, but without management it becomes a "data swamp" no one can use. On AWS, this is S3 + Glue Data Catalog + Athena/Lake Formation.

Data warehouse: schema-on-write
  Validate schema before loading → clean but rigid

Data lake: schema-on-read
  Store everything first → apply schema at read time → flexible but needs management

These days the Lakehouse pattern, which combines the strengths of both, is the standard. It layers a table format (such as Apache Iceberg) and a catalog on top of an S3 data lake to provide warehouse-grade queries and transactions.

💡 Related theory: "Schema-on-write vs schema-on-read" is the essence of what separates the lake from the warehouse. The warehouse enforces a schema at load time, guaranteeing data quality but making it fragile to change. The lake accepts any shape at load time, so it can prepare for unknown future analyses — but without a catalog and governance it becomes ungovernable. AWS Lake Formation is precisely the service that aims to fill this governance gap.

What DEA-C01 Asks

The exam splits the data pipeline lifecycle into four domains.

The largest domain, Ingestion and Transformation (34%), shows that this is the heart of a data engineer's work. Where SAA asks "how do you design the architecture," DEA asks "how do you ingest, store, transform, operate, and safely govern data."

Wrapping Up

We drew three pictures today. First, a data engineer's deliverable is a trustworthy pipeline, and the number-one responsibility is data quality. Second, OLTP (transactions) and OLAP (analytics) have opposite workloads, so their storage and query approaches differ. Third, the data lake (schema-on-read) and the warehouse (schema-on-write) have different storage philosophies, and the lakehouse that combines them is the modern standard.

In the next article, we look at the two processing paradigms that split based on "when and how much" data arrives — batch and streaming.