Day 3 - A Bird's-Eye View of AWS Data Services

📌 Key Takeaways

• AWS data services are laid out along the ingest → store → process → analyze flow, and governance (catalog, permissions, encryption) wraps the entire span.
• Scenario questions first ask "which stage of the flow is this?", then narrow the candidates for that stage by source type, freshness, and operational overhead.
• The source decides ingestion: events = Kinesis/MSK, database changes = DMS (CDC), files = DataSync, SaaS = AppFlow.
• S3 is the default for storage, and the Glue Data Catalog is the common metadata that Athena, Redshift Spectrum, and EMR all look at together.
• Permissions work at different layers: IAM handles resources (buckets, APIs), Lake Formation handles tables, columns, and rows, and KMS handles encryption keys.

The Data Flow Is the Service Map

AWS has dozens of data-related services. When you first open the console, a flood of similar-looking names such as Kinesis, Glue, EMR, Redshift, Athena, and Lake Formation leaves you at a loss. But if you line them up in the order the data flows, a map emerges. Data always follows the flow of ingest → store → process → analyze, and governance (security and catalog) wraps across all of it.

Today we draw a map of which services belong where across these five categories. We will cover the deep details of individual services in later weeks; today's goal is to place in your head "which stage of the pipeline is this service in?" A good share of exam questions ask "which service fits this scenario?", and the answer comes from this map.

This single picture is the skeleton of DEA-C01 service knowledge. Data flows from the left (ingestion) to the right (analytics), and the governance layer at the bottom oversees permissions, catalog, and encryption across the whole span.

💡 Related theory: In industry-standard terms, this flow is called the "stages of a data pipeline." Each stage should be loosely coupled. For example, if you place S3 or a stream buffer between ingestion (Kinesis) and processing (Lambda), ingestion does not get blocked even when the processing stage is slow. This pattern of "separating stages with a buffer" is the heart of robust pipeline design.

Ingestion: Bringing Data In

This is the entrance for bringing data from a source into AWS. The tool differs depending on the nature of the source.

Key distinctions: real-time event streams go to Kinesis/MSK, to bring in the changes of an existing database use DMS (including CDC), and large chunks of on-premises files go to DataSync.

The real value of DMS is in CDC (Change Data Capture). It reads the transaction log of the production DB and continuously replicates "only the changes," so you keep the analytics environment current without dumping the whole thing each time. It is the standard way to flow data to OLAP while putting almost no load on OLTP.

Storage: Holding Data

This is where the ingested data stays. The OLTP/OLAP and lake/warehouse distinctions from Day 1 apply as they are.

S3 is at the center of everything. It is the foundation of the data lake and the input/output point of almost every processing and analytics service. When "where do I store it?" is vague, S3 is the default.

# S3 is the foundation of the data lake — store using a partition structure
aws s3 cp orders.parquet \
  s3://datalake/raw/orders/year=2026/month=06/day=25/
 
# Athena queries this S3 location directly with SQL (no separate loading needed)
# If the Glue Catalog knows the schema, you can query immediately

S3 storage classes (Standard, Intelligent-Tiering, Glacier, etc.) and lifecycle policies are the key to cost optimization. Automatically moving rarely used past data to cheaper tiers is the everyday work of a data engineer, and it is covered with real weight on the exam as well.

Processing: Transforming Data

These are the engines that cleanse and transform raw data into an analyzable form. You choose by data volume and the nature of the job.

Glue is the "default for managed ETL." It runs Spark-based transformations without server management, and the Data Catalog, crawlers, and scheduler are integrated. EMR is used when you need more control and scale, or when you move existing Hadoop/Spark code over as is. Lambda suits small-unit, event-driven transformations (such as processing when a single file arrives).

A Glue Crawler scans S3 data, automatically infers the schema, and registers it as a table in the Glue Data Catalog. This catalog becomes the "common metadata dictionary" shared by Athena, Redshift Spectrum, and EMR. In other words, once registered in the catalog, multiple analytics engines see the same data with the same definition.

Analytics: Getting Answers from Data

This is the consumption layer that asks questions of the refined data.

Athena is a serverless tool that queries data loaded in S3 with SQL right where it sits, without moving it. There is no infrastructure and you are billed by the amount of data scanned, so it is ideal for occasional ad hoc analysis. If you query frequently, quickly, and at large scale, load the data into Redshift. To turn the final results into a dashboard that people look at, use QuickSight.

Governance: Controlling the Whole Span

This is the layer of security, catalog, and permissions that cuts across every stage from ingestion to analytics.

Lake Formation grants fine-grained permissions at the "table, column, and row level" on the S3 data lake. Where IAM asks "can this person access this S3 bucket?", Lake Formation controls "can this person see up to this column of this table?"

The reason governance is a separate category is that in a data pipeline, security, permissions, and lineage are not a problem of one stage but a cross-cutting concern that runs through the entire span. The essence of governance is tracking who accessed which data, and where that data came from and where it went (data lineage).

Pieces Not Yet on the Map

The table above contains only the skeleton. The DEA-C01 scope includes more services that connect the stages or fill gaps. When you see a name, you should immediately recall "what goes in which box."

• Orchestration is not the ingestion or processing stage itself but the layer that decides when and in what order those stages run. Glue workflows and triggers also belong here, but they are used only when tying together Glue jobs alone.
• The Kinesis Data Firehose in the tables above is now named Amazon Data Firehose. Both spellings appear in exam passages.

Lookalike Service Decision Table

What tends to become a wrong answer in this track is the option where "both seem like they would work." Remember just one axis along which they split.

Ingestion: what is the source?

• DMS moves rows and DataSync moves files. "Copy a database backup file to S3" is a file transfer, so it is DataSync; "keep a database's tables continuously synchronized" is DMS.

Processing: how big, and how much control is needed?

Analytics: where does the data live and how often is it queried?

Governance: what does it control?

⚠️ Pitfall: An option that tries to solve "column-level access control" with only an S3 bucket policy or an IAM policy is a wrong answer. S3 only knows down to the object level. What hides columns and rows inside an object is Lake Formation, which applies permissions to catalog tables.

What the Glue Data Catalog Actually Does

The catalog does not hold the data. It holds only the instruction manual: "where in S3 is this table's data, what are its columns and types, and in what format should it be read?"

-- Athena DDL: the data stays in S3 as is; only the manual is registered in the catalog
CREATE EXTERNAL TABLE orders (
  order_id   string,
  amount     double,
  region     string
)
PARTITIONED BY (dt string)
STORED AS PARQUET
LOCATION 's3://datalake/processed/orders/';
 
-- When a new date folder appears, you must tell the catalog about the partition for it to be queryable
ALTER TABLE orders ADD IF NOT EXISTS
  PARTITION (dt = '2026-06-25')
  LOCATION 's3://datalake/processed/orders/dt=2026-06-25/';

• The hierarchy is database → table → partition. A table points to the S3 path, format (SerDe), and columns, and a partition points to a sub-path.
• Even if you delete an external table with DROP TABLE, the files in S3 remain. Conversely, even if you delete the S3 files, the catalog entry remains and you get empty results.
• There are three ways to populate the catalog.

• A crawler costs money and time every time it runs. The cause of "a new file came in but I can't see it in Athena" is usually that the partition is not registered.
• MSCK REPAIR TABLE only adds new partitions, takes a long time, and cannot clean up deleted partitions. For partition management, ALTER TABLE ADD PARTITION, crawlers, and partition projection are better.

That several engines share one catalog is both a benefit and a responsibility. If someone changes a table's column type, not only Athena queries but also the Glue jobs, EMR jobs, and Redshift Spectrum external schemas that read that table are all affected at once. That is why Lake Formation manages not only the permission to query data but separately the permission to modify catalog metadata.

The Billing Method Tells You the Nature of the Service

The exam does not ask about amounts but about the structure. If you know "what the cost is proportional to," you can pick out the "most cost-effective" option.

• For occasional use, usage-based billing (Athena, Lambda, on-demand/serverless) is advantageous, and for continuous, fully loaded use, capacity-based billing (provisioned clusters and shards) is advantageous. This single line is the criterion for "serverless vs. provisioned" questions.

Scenario → Service Decision Table

In the next post, we look at what "form" the data flowing through this pipeline is stored in — CSV, JSON, Parquet, ORC, Avro, columnar vs. row-oriented, and schema evolution.

📖 Glossary

• Ingestion : The stage of bringing source system data into AWS. The tool differs depending on the type of source.
• CDC (Change Data Capture) : A method of reading a database's change log and continuously replicating only the changed rows.
• Data catalog : An instruction manual recording where data is and what its columns and formats are. It does not hold the data itself.
• Crawler : A Glue feature that scans a store, infers the schema and partitions, and registers them as tables in the catalog.
• External table : A table where the data stays in S3 and only the definition is registered. Deleting the table leaves the files.
• Partition : A unit that splits an S3 path by a value such as a date. It makes queries read only the folders that match the condition, reducing the scan amount.
• Serverless : A way of paying only for what you use without launching servers or clusters yourself. Athena, Glue, and Lambda qualify.
• Orchestration : Automatically chaining and running multiple jobs according to rules of order, conditions, and retries.
• Data lineage : The history of where data came from, what processing it went through, and where it went.
• Ad hoc query : A query thrown on the spot when needed, not decided in advance.