Day 3 - A Bird's-Eye View of AWS Data Services

AWS has dozens of data-related services. Open the console for the first time and Kinesis, Glue, EMR, Redshift, Athena, Lake Formation… a flood of similar-sounding names leaves you overwhelmed. But line them up in the order data flows, and a map emerges. Data always follows the flow of ingestion → storage → processing → analytics, with governance (security and cataloging) wrapping around the whole thing.

Today we draw the map of which services belong to which of these five categories. Deep details of individual services come in later weeks; today's goal is to place "which stage of the pipeline does this service live in" in your head. Half the exam questions are "which service fits this scenario?", and the answer comes from this map.

The Data Flow Is the Service Map

[Ingestion]  [Storage]     [Processing]    [Analytics]
 Kinesis  →  S3       →   Glue        →  Athena
 MSK         Redshift     EMR            Redshift
 DMS         DynamoDB     Lambda         QuickSight
 DataSync                 EMR Serverless OpenSearch
   │            │            │               │
   └────────────┴────────────┴───────────────┘
         [Governance] Lake Formation · Glue Catalog · IAM · KMS

This single diagram is the skeleton of DEA-C01 service knowledge. Data flows from left (ingestion) to right (analytics), and the governance layer at the bottom oversees permissions, cataloging, and encryption across every stage.

💡 Related theory: In industry-standard terminology, this flow is called "the stages of a data pipeline." Each stage should be loosely coupled. For example, placing S3 or a stream buffer between ingestion (Kinesis) and processing (Lambda) means ingestion doesn't get blocked even when the processing stage slows down. This pattern of "decoupling stages with buffers" is the key to designing robust pipelines.

Ingestion: Bringing Data In

This is the entry point that brings data from the source into AWS. The tool depends on the nature of the source.

The key distinctions: real-time event streams go to Kinesis/MSK, pulling changes from an existing database means DMS (including CDC), and bulk on-prem files mean DataSync.

💡 Related theory: DMS's real value lies in CDC (Change Data Capture). It reads the operational DB's transaction log and continuously replicates "only the changes," keeping the analytics environment current without dumping everything each time. This is the standard way to flow data into OLAP while putting almost no load on OLTP.

Storage: Holding Data

Where the ingested data lives. Day 1's OLTP/OLAP and lake/warehouse distinctions apply directly.

S3 is the center of everything. It is the foundation of the data lake and the input/output point of nearly every processing and analytics service. When "where should I store this" feels vague, S3 is the default.

# S3 is the foundation of the data lake — store in a partitioned structure
aws s3 cp orders.parquet \
  s3://datalake/raw/orders/year=2026/month=06/day=25/
 
# Athena runs SQL queries directly on this S3 data (no separate loading needed)
# If the Glue Catalog knows the schema, it's instantly queryable

💡 Related theory: S3 storage classes (Standard, Intelligent-Tiering, Glacier, etc.) and lifecycle policies are the heart of cost optimization. Automatically moving infrequently used historical data to cheaper tiers is a data engineer's day-to-day work, and the exam covers it with real weight.

Processing: Transforming Data

These are the engines that cleanse and transform raw data into an analyzable form. Choose by data volume and the nature of the job.

Glue is the "default for managed ETL." It runs Spark-based transformations without server management, with an integrated data catalog, crawlers, and scheduler. EMR is for when you need more control and scale, or when porting existing Hadoop/Spark code as-is. Lambda suits small event-driven transforms (e.g., processing a file the moment it arrives).

💡 Related theory: A Glue Crawler scans S3 data, automatically infers the schema, and registers it as a table in the Glue Data Catalog. This catalog becomes the "shared metadata dictionary" used by Athena, Redshift Spectrum, and EMR. In other words, register once in the catalog, and multiple analytics engines see the same data with the same definitions.

Analytics: Getting Answers from Data

The consumption layer where you pose questions to the cleansed data.

Athena queries data loaded in S3 with SQL right where it sits, without moving it — a serverless tool. With no infrastructure and billing based only on the amount of data scanned, it's ideal for occasional ad-hoc analysis. If you query frequently, quickly, and at large scale, load the data into Redshift. To turn final results into dashboards for humans, it's QuickSight.

Governance: Controlling the Entire Span

The security, catalog, and permissions layer that cuts across every stage from ingestion to analytics.

Lake Formation grants fine-grained "table, column, and row level" permissions on the S3 data lake. If IAM answers "can this person access this S3 bucket," Lake Formation controls "can this person see this column of this table."

💡 Related theory: Governance is its own category because in a data pipeline, security, permissions, and lineage are not a single stage's problem but a cross-cutting concern that runs through the entire span. Tracking who accessed which data, and where that data came from and went (data lineage), is the essence of governance.

Wrapping Up

The map we drew today boils down to one thing. AWS data services are arranged along the flow of ingestion → storage → processing → analytics, with governance wrapping around the whole. When you face a vague scenario question, first ask "which stage of the flow is this," then narrow the candidate services in that stage by "real-time or batch, managed or full control, and how much data."

In the next article, we look at what "shape" the data flowing through this pipeline is stored in — CSV, JSON, Parquet, ORC, Avro, columnar vs row-oriented, and schema evolution.