Based on the SAP-C02 exam guideReport an error
Engineers designing big data processing for the first time typically start by asking, "Which service should we use? EMR? Glue? Athena?" But these three aren't competitors—they're the same engine (Apache Spark) wrapped in different operational models. EMR lets you directly operate Spark as a cluster; Glue abstracts Spark serverlessly; Athena exposes only SQL on top of Spark/Presto. So the real question becomes: "How much direct control of the cluster does this workload need, and how much operational burden are we willing to offload to AWS?"
In SAP-C02 exams, this domain tests operational architecture: "running heavy Spark jobs across tens of thousands of nodes processing petabytes cost-optimally," "orchestrating pipelines with hundreds of interdependent steps," "keeping jobs alive through Spot interruptions and node failures." Today we'll dissect how EMR, Glue, and MWAA work internally, why they're architected that way, and build intuition for solving these scenarios.
Big data processing originated with Google's 2004 MapReduce paper. The core idea was simple: split vast data into chunks, scatter them across thousands of commodity servers (Map), have each independently process their chunk, then combine results (Reduce). Linear scaling from one supercomputer to thousands of cheap servers. Doug Cutting implemented this as open-source Hadoop—HDFS for distributed storage (Google GFS's open-source version), YARN for resource management.
The problem: MapReduce wrote all intermediate results to disk. Iterative algorithms (machine learning, graph processing) cycle over the same data dozens of times; writing to/reading from HDFS each cycle created an I/O bottleneck. In 2009, Matei Zaharia at UC Berkeley's AMPLab created Spark, inverting this. Spark keeps intermediate results in memory (in-memory computing) and expresses computation as RDD (Resilient Distributed Dataset), an immutable abstraction. RDDs don't hold actual data—only the "lineage" of how they're built. If a node dies and some partitions vanish, Spark recomputes just that portion along the lineage, restoring it. This fault-tolerance without checkpointing made Spark the standard.
💡 Related Theory: Spark's core is lazy evaluation and DAG execution. Transformations like
mapandfilterdon't execute immediately—only recorded in the lineage. Only when an action (count, collect, save) runs does Spark see the whole transformation chain, build an optimal plan (DAG), and execute it once. This enables operator fusion (merging unnecessary steps) and shuffle minimization. With DataFrame API, you add the Catalyst optimizer and Tungsten execution engine, so declaratively-written SQL-like code gets column pruning, predicate pushdown, and code generation, running far faster than RDD. So practically and on exams, "DataFrame/Spark SQL over RDD" is standard.
🔍 Deep Dive: Shuffle is the most expensive operation in distributed processing because it is why. Wide transformations (
groupBy,join,reduceByKey) must group equal keys on the same node. This triggers all-to-all network communication between every node, writes intermediate results to disk (shuffle write), then reads them (shuffle read). Slow 100-million-row joins usually stem from shuffle, not computation itself. Optimization hinges on reducing shuffle—small tables use broadcast join (replicate to all nodes, kill shuffle), pre-align partitioning for co-location, avoidrepartitionoveruse. Redshift's distribution key (Day 42) follows the identical principle: minimize inter-node shuffle.
EMR (Elastic MapReduce) runs the Hadoop ecosystem (Spark, Hive, Presto/Trino, HBase, Flink, Hudi) on EC2 clusters as a managed service. It eliminates hand-installing, tuning, and patching Hadoop on EC2; clicks or one API call spin up dozens to thousands of nodes in minutes. Understanding EMR hinges on three node roles.
This role separation starts EMR cost optimization. Keep Master/Core minimal on stable On-Demand; elastically scale heavy compute with cheap Spot Task nodes.
💡 Related Theory: Where EMR places data determines architecture. Traditional Hadoop pursues data locality—execute computation where data lives (on Core nodes' local HDFS). In cloud, EMRFS becomes standard: data lives on S3. This separates compute and storage (like RA3 on Day 42), letting jobs terminate clusters entirely post-run (zero compute cost) and multiple clusters share the same S3 data. Downside: S3 has higher latency than HDFS, but EMRFS's S3 optimization committer and Parquet column pruning offset this considerably. From SAP's angle: "kill cluster post-job to cut costs" almost always means S3-based transient cluster.
🔍 Deep Dive: Instance Fleets vs Instance Groups. Two ways to configure EMR nodes. Instance Groups pin each group to a single instance type—simple but fails when that type's Spot inventory dries up. Instance Fleets mix multiple instance types (m5.xlarge, m5a.xlarge, m4.xlarge) in one group with weights and target capacity (e.g., "fill 32 vCPU"). EMR auto-fills with the cheapest available, auto-substituting if one type's Spot interrupts. So "Spot-resilient, cost-optimal" in exams almost always means Instance Fleets + Spot multi-type.
📚 Case Study: Yelp, an early large-scale EMR operator, ran hundreds of clusters daily, ingesting petabytes of logs via Spark/MapReduce. Core pattern: "transient cluster per job"—data lives permanently on S3; only when processing does a cluster spin up, fill Task nodes with Spot, then terminate immediately, minimizing compute costs. Lessons: (1) transient per-job clusters beat long-running for cost/isolation, (2) S3 as source of truth makes clusters disposable. SAP exams repeat this "cost-optimal batch" pattern.
EMR offers the same engine in three runtimes. Choice reflects "control vs. operational burden" tradeoff.
⚠️ Trap: "EMR Serverless is always cheaper and better"—false. Serverless bills per vCPU-memory-hour of worker runtime; sustained workloads can cost more than well-tuned EC2 clusters (especially Spot Fleets). Serverless trades lower operational burden for fractionally higher unit cost. Long-running, 24-hour steady workloads may favor EC2. Components like HBase need long-lived clusters; Serverless can't run them. In exams: "sporadic, variable, ops-averse" = Serverless; "sustained, 24/7, special components" = EC2.
Decision criteria in one table. Core variables: startup, ops burden, workload weight.
| Aspect | EMR (on EC2) | Glue | Athena |
|---|---|---|---|
| Engine | Full Hadoop/Spark/Hive/Presto/HBase | Serverless Spark (+ Python Shell, Ray) | Presto/Trino (SQL), Spark (analytics) |
| Ops Model | Direct cluster management | Serverless (DPU) | Fully serverless |
| Startup | 5–10 min (cluster boot) | ~1 min (warm pool: seconds) | Instant |
| Billing | Cluster uptime (EC2) | DPU-hours | S3 scan volume |
| Best For | Heavy repeat, custom tuning, special components | Light–medium ETL, catalog integration | Ad-hoc SQL, BI |
| Control | Maximum | Medium (abstracted) | Minimal (SQL only) |
Decision flow: "SQL sufficient and ad-hoc" = Athena. "Structured ETL but no cluster ops" = Glue. "Heavy repeat, custom Spark tuning, full Hive/HBase/Flink stack needed" = EMR. All three share Glue Data Catalog as common metastore, so the same table definition appears across all three engines.
🔍 Deep Dive: How Glue Job Bookmark enables incremental processing. Daily ETL that reprocesses yesterday's files wastes cost and duplicates results. Job Bookmark stores "how far we got" state (S3 object last-modified time/path, JDBC source column value) per run as checkpoint. Next run processes only new data after that mark. It's the streaming-offset-management idea adapted for batch. In exams, "daily Glue job reprocesses already-done data / want incremental only" signals Job Bookmark enable. (Antipattern: enable bookmark but write job logic that reads everything from scratch anyway—pointless.)
Glue isn't just ETL jobs; it's a data integration platform. Glue Data Catalog is a Hive Metastore-compatible metadata repository holding table schemas, partitions, locations—shared by Athena, EMR, Redshift Spectrum, and Glue (the true hub here). Glue Crawler scans S3 and JDBC sources, infers schemas, auto-registers tables/partitions in the catalog. Glue DataBrew is a code-free visual tool to craft data-cleaning/transformation recipes, letting data analysts self-serve cleaning without engineers.
💡 Related Theory: Glue Data Catalog's Hive Metastore compatibility is the ecosystem's linchpin. Hive Metastore (2008) standardized "what columns a table has, what file format/path" in Hadoop. AWS implemented this interface precisely, so the same Parquet file in S3 can be SQL-queried via Athena, transformed via EMR Spark, and joined via Redshift Spectrum—all as "the same table." Centralizing metadata and swappable engines builds the lakehouse architecture's foundation.
Data pipelines flow ingest → store → process → ingest → analyze, with many steps holding dependencies. "Glue job must finish before EMR starts, then Athena CTAS runs, retry on failure, trigger at specific time." Managing dependencies, scheduling, retries, and observability is orchestration; the de facto standard is Apache Airflow. MWAA (Managed Workflows for Apache Airflow) is AWS's managed Airflow, handling webserver, scheduler, workers.
Airflow's core model is DAG (Directed Acyclic Graph). Workflows express as "acyclic directed graphs," explicitly stating step dependencies. No cycles matter because A→B→A deadlocks forever. DAGs are Python code, so conditional branches, dynamic task generation, and external integrations flow from code's expressiveness. Rich Operator set (EMR, Glue, Athena, Step Functions, etc.) weaves multi-service pipelines in one DAG.
💡 Related Theory: DAG is computer science's universal abstraction for dependency and scheduling. Build systems (Makefile, Bazel), package resolution, Spark execution plans, even Git commit graphs are DAGs. The core operation is topological sort—"order as: 'run tasks whose prerequisites finished'." Airflow's scheduler does exactly this every moment: find tasks where upstream succeeded (dependencies met), queue them for execution. Cycles break topological sorts, hence "Acyclic" in DAG.
AWS has two orchestration tools: MWAA (Airflow) and Step Functions. Different philosophies.
| Aspect | MWAA (Airflow) | Step Functions |
|---|---|---|
| Definition | Python DAG (code) | ASL (Amazon States Language) JSON |
| Integration | Operator ecosystem (diverse external) | 200+ AWS services natively |
| Ops | Airflow environment mgmt (managed but env cost/version) | Fully serverless, zero ops |
| Portability | Airflow OSS—on-prem, multi-cloud migration | AWS-only |
| Billing | Environment uptime (always-on cost) | State transition per execution |
| Strength | Complex data pipelines, rich community | Event-driven, serverless workflows, deep AWS integration |
Pick MWAA if: already using Airflow, need multi-cloud portability, complex Python logic, rich data Operators. Pick Step Functions if: pure AWS, serverless zero-ops, event-driven, no always-on cost. In exams: "Python DAG / Airflow migration / portability" = MWAA; "serverless / AWS deep integration / zero ops / state-transition billing" = Step Functions.
⚠️ Trap: MWAA is "managed" but not free. Airflow environment (webserver, scheduler, minimum worker) runs always, costing per-hour whether or not jobs run. Step Functions bill per state transition; no workflow execution = cost near zero. "Sporadic workload, minimize always-on cost" signals Step Functions (or EventBridge + Step Functions), not MWAA. "Complex data DAG with existing Airflow assets" is MWAA's legitimate domain.
This domain builds on open-source standards (Spark, Airflow, Kafka, Hive), so multi-cloud mappings are clear.
| Role | AWS | GCP | Azure |
|---|---|---|---|
| Managed Spark/Hadoop cluster | EMR | Dataproc | HDInsight / Synapse Spark |
| Serverless ETL | Glue | Dataflow (Beam) / Dataproc Serverless | Data Factory / Synapse |
| Serverless SQL on object storage | Athena | BigQuery (external tables) | Synapse Serverless SQL |
| Managed Airflow | MWAA | Cloud Composer | Data Factory Managed Airflow |
| Metadata Catalog | Glue Catalog | Dataplex / Data Catalog | Purview |
Core insight: all these services wrap the same open-source (Spark, Airflow, Hive Metastore), so maintaining catalog (metadata) standards lets you swap processing engines relatively freely. Multi-cloud and hybrid strategies standardize on Airflow (MWAA/Composer) and Iceberg/Hive Metastore-compatible catalogs.
EMR, Glue, Athena are not competitors but the same Spark/Presto engine wrapped differently on a control-vs-burden axis. EMR: full control (cluster). Glue: serverless ETL (DPU). Athena: SQL (scan billing). EMR cost optimization's heart: fill Task nodes with Spot, use Instance Fleets for Spot-interruption resilience, keep data on S3 (EMRFS) and operate transient clusters. MWAA (Airflow DAG) and Step Functions (ASL) orchestrate pipelines; former emphasizes portability, complex logic, and persistent assets; latter emphasizes serverless, AWS integration, zero ops.
SAP exam recurring mappings: (1) "EMR Task cost optimal + Spot resilient" → Instance Fleets + Spot multi-type. (2) "Kill cluster post-job to cut costs" → S3 (EMRFS) transient cluster. (3) "Variable Spark, no cluster ops" → EMR Serverless. (4) "Light ETL, no cluster ops" → Glue. (5) "Process incremental only" → Glue Job Bookmark. (6) "Python DAG, portability" → MWAA. (7) "Serverless, AWS integration, zero ops, zero always-on cost" → Step Functions. (8) "Ad-hoc SQL" → Athena. Next day covers data governance (Lake Formation) and real-time streaming (MSK).
Click a choice to reveal the answer and explanation.
Question 1
You batch-process petabyte-scale logs with Spark every day. The data is retained permanently in S3, and processing takes only 2-3 hours a day. How can you minimize cost while making sure jobs do not die from Spot interruptions?
Question 2
There is a Spark job that runs a few times a day at irregular times. You do not want to size clusters or manage nodes, and you want to minimize cost when it is not running. What should you use?
Question 3
A daily Glue ETL job reprocesses S3 files it already processed yesterday, raising cost and duplicating results. How can you make it process only newly arrived data?
Question 4
You need to orchestrate a data pipeline in which dozens of steps have dependencies. The team already uses Apache Airflow on premises and wants portability, considering a possible move to another cloud later. What is the most appropriate choice?
Question 5
You are building an event-driven workflow in a pure AWS environment. It runs only intermittently, you want no standing cost when the workflow is not running, and it must integrate closely with many AWS services. What is the most appropriate choice?
Question 6
Data analysts want to cleanse and transform data visually without help from engineers. Which tool lets them build cleansing recipes without writing code?
Question 7
An organization has already standardized its container platform on EKS and wants to integrate application workloads and big data Spark workloads onto the same Kubernetes resource pool, IAM, and observability tools to raise resource utilization. What is the most appropriate EMR deployment model?