Day 1 - Amazon EMR: Spark, Hive and Cluster Operations, Plus EMR Serverless

Until last week, Glue represented "ETL without worrying about servers." But when data scales into terabytes, when you need complex ML preprocessing or big-data ecosystem tools like Hive, HBase, and Presto, you need greater freedom and control. That's where Amazon EMR (Elastic MapReduce) comes in.

EMR is, simply put, a "managed Hadoop/Spark cluster." It automatically provisions and operates open-source big-data frameworks — Apache Spark, Hive, Presto, HBase, Flink — bundled on EC2. If Glue is "an abstracted ETL function," EMR is more like "a distributed computing cluster you operate directly." On exams, the selection criteria between these two are always a hot topic.

EMR Cluster Structure: Three Node Roles

An EMR cluster is a collection of nodes (EC2 instances), each with a role.

The key distinction: Core nodes hold HDFS data, Task nodes only compute. This difference directly affects cost strategy. Task nodes don't store data, so they can disappear without data loss. Therefore, using spot instances on Task nodes is the safe standard pattern. Conversely, if you put Core nodes on spot, node reclamation might lose HDFS blocks, requiring recalculation.

💡 Related Theory: EMR's node distinction comes from Hadoop's MapReduce architecture. Hadoop originally followed the data locality principle: perform computation on nodes where data lives — "move compute to data." Moving massive data over the network costs far more than sending small code to where data resides. Core nodes handling both storage and compute trace this principle.

Spark on EMR: In-Memory Distributed Processing

The most widely used engine on EMR is Apache Spark. Spark's essence is keeping intermediate results in memory, not disk. Unlike traditional MapReduce, which writes and reads intermediate results to HDFS at each stage, Spark processes in memory—making it tens of times faster for iterative operations (ML, graphs).

# EMR Spark: Read from S3, transform, save as Parquet
from pyspark.sql import SparkSession
 
spark = SparkSession.builder.appName("daily-etl").getOrCreate()
 
df = spark.read.json("s3://raw-bucket/events/2026/06/26/")
result = (df
    .filter(df.amount > 0)
    .groupBy("region", "product_id")
    .agg({"amount": "sum"})
    .withColumnRenamed("sum(amount)", "total_sales"))
 
result.write.mode("overwrite").parquet("s3://curated-bucket/sales/")

Spark's speed also comes from lazy evaluation. Transformations like filter and groupBy aren't executed immediately; they accumulate as an execution plan (DAG). Only when an action like write or count is called does Spark optimize the entire DAG and run it once. This lets you merge or skip unnecessary intermediate steps.

🔍 Deeper Dive: Spark's underlying data structure is the RDD (Resilient Distributed Dataset). "Resilient" because each RDD remembers its lineage — how it was created. If a node dies and some partitions vanish, you can recalculate just those partitions following the lineage. No need to checkpoint everything; faults are tolerated smartly. DataFrame and SQL are higher-level abstractions layered on RDD.

EMR Storage: HDFS vs EMRFS

EMR can use two storage tiers.

Modern EMR best practice: use EMRFS (S3) as permanent storage. With data on S3, you can spin up the cluster when needed and tear it down when done (transient cluster). Compute and storage separate—no need to keep the cluster running 24/7, just pay when working. HDFS is only for performance-critical temporary storage like shuffle intermediate results or repeated operation scratchpad.

⚠️ Gotcha: If you think "EMR = permanently store data in HDFS," you'll fail exams. When you terminate a transient cluster, HDFS disappears. Permanent data must go to S3 (EMRFS); this is the core of cost optimization.

Cluster Purchase Options: On-Demand vs Spot vs Instance Fleets

Cost is half of EMR exam questions. The key is which pricing model to use for nodes.

• On-Demand: Full price, never reclaimed. Suitable for Primary/Core nodes needing stability.
• Spot instances: Up to 90% cheaper but AWS can terminate within 2 minutes if capacity is needed. Perfect for Task nodes.
• Reserved/Savings Plans: Cost savings for long-term always-on clusters.

EMR offers two ways to define node groups.

Instance Fleets lets you list multiple candidates: "if m5.xlarge spot is unavailable, use m5.2xlarge or c5.xlarge instead." This resilience to spot reclamation is the modern recommended approach for spot-heavy clusters.

🎯 Scenario: A large batch ETL runs only 1-2 hours at 2 AM daily. Cost-optimal setup: (1) Transient cluster spins up at start, auto-terminates at end, (2) data permanently on S3 (EMRFS), (3) Primary and Core nodes on-demand for stability, (4) Task nodes Instance Fleets + spot for cheap compute scaling. Submit work via EMR Steps, auto-terminate cluster on last step completion.

EMR Serverless: Eliminate the Cluster Itself

If deciding cluster size, choosing node types, and planning spot strategy sounds burdensome, EMR Serverless is the answer. No cluster provisioning—submit a Spark or Hive app and EMR auto-provisions workers, runs it, and reclaims them. You pay only for vCPU, memory, and time used.

# Submit a Spark job to EMR Serverless app
aws emr-serverless start-job-run \
  --application-id 00abc123 \
  --execution-role-arn arn:aws:iam::111122223333:role/emr-serverless-role \
  --job-driver '{
    "sparkSubmit": {
      "entryPoint": "s3://scripts/daily-etl.py"
    }
  }'

EMR Serverless suits sporadic or hard-to-predict workloads, and when you don't want to spend time sizing and tuning clusters. Conversely, if you need resident services like HBase or must keep clusters on 24/7 for fine-tuning, traditional EMR on EC2 is right.

When Glue, When EMR?

This is the exam's critical decision point.

In summary: Glue is simple, fast, serverless ETL; EMR is scale, control, framework diversity. If "minimal operations overhead" is emphasized, lean Glue. If "cost control and big-data framework choice" is emphasized, lean EMR.

Summary

EMR is a managed big-data cluster where the Core/Task node distinction is key to spot cost strategy. Store data permanently on S3 (EMRFS) to separate compute from storage, use transient clusters to cut costs. If cluster operations seem burdensome, use EMR Serverless. Glue vs EMR is a balance between "operational simplicity" and "control and scale." Tomorrow we cover a lighter transform tool: Lambda.