Day 1 - Data Transformation and ETL: AWS Glue, Spark, and EMR

📌 Key Takeaways

• ETL = Extract → Transform → Load. In ML, the output is the training dataset itself, so the correctness and reproducibility of the transformation logic determine model quality.
• Glue = serverless Spark. Low operational burden and suited to sporadic, standard ETL. Billed in DPU-hours.
• EMR = Hadoop/Spark/Hive/Presto on an EC2 cluster. Use it when you need fine-grained tuning, multiple frameworks, or Spot savings.
• The four storage strategies for large-scale preprocessing: columnar formats, partitioning, file size optimization, and pushdown.
• Division of roles: shared cleansing across the data lake = Glue, preprocessing dedicated to a specific model = SageMaker Processing.

What Is ETL?

Last week we covered "where and how to bring the data in" (ingestion and storage). Now comes the stage where you turn that raw data into a form the model can consume. On MLS-C01, ETL is a core area that asks "which tool do you choose, at what scale, with what cost and operational burden?"

A variant called ELT (Extract-Load-Transform) is also used these days. You load raw data into the data lake (S3) first and transform it when needed. The data lake + Athena/Spark combination is effectively an ELT pattern, and it works well when you handle large volumes of data that are less structured.

Transformations You Actually Do in ML Preprocessing

AWS Glue — The Center of Serverless ETL

AWS Glue is an Apache Spark-based ETL service where you do not manage servers yourself (serverless). AWS launches and shuts down the cluster for you, so engineers focus only on the transformation logic.

• Glue Data Catalog — A central metadata store holding the schema, location, and partition information of your data. Shared by Athena, EMR, and Redshift Spectrum.
• Glue Crawler — Scans data in S3 and elsewhere, infers the schema automatically, and registers tables in the Data Catalog.
• Glue ETL Job — The Spark (or Python Shell) job that performs the actual transformation. Written in PySpark or Scala.
• DynamicFrame — Glue's extension of the Spark DataFrame. Useful for handling data with irregular schemas.
• Glue DataBrew — A separate tool for building cleansing recipes visually without code. The no-code option for data preparation.

# Skeleton of a Glue PySpark ETL job
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
 
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
 
# 1) Read raw data from the Data Catalog (Extract)
raw = glueContext.create_dynamic_frame.from_catalog(
    database="ml_raw", table_name="clicks"
)
 
# 2) Transform: drop null fields + map columns (Transform)
mapped = ApplyMapping.apply(frame=raw, mappings=[
    ("user_id", "string", "user_id", "string"),
    ("ts", "string", "event_time", "timestamp"),
    ("amount", "string", "amount", "double"),
])
clean = DropNullFields.apply(frame=mapped)
 
# 3) Load into the training S3 bucket as Parquet (Load)
glueContext.write_dynamic_frame.from_options(
    frame=clean,
    connection_type="s3",
    connection_options={"path": "s3://ml-train/clicks/"},
    format="parquet"
)
job.commit()

A Glue job's processing capacity is measured in DPUs (Data Processing Units). 1 DPU is roughly 4 vCPU + 16GB of memory. If a job is slow, you scale out horizontally by increasing the number of DPUs (workers), or you raise performance by using the Spark engine in a newer Glue version. You are billed only for the DPU-hours you actually use.

Amazon EMR — A Full-Control Big Data Cluster

EMR (Elastic MapReduce) is a managed service for running big data frameworks such as Hadoop, Spark, Hive, Presto, and HBase directly on an EC2 cluster. If Glue is "Spark only, serverless, simple," EMR is the option for "multiple engines, fine-grained tuning, direct control."

Situations where EMR is advantageous:

• When you process petabyte-scale data for long periods and need fine-grained cluster tuning (instance types, memory, executor settings)
• When you need to use a variety of frameworks such as Hive, Presto, and HBase alongside Spark
• When you want to cut costs substantially with Spot Instances (workloads that tolerate interruption)

# Create an EMR cluster (with Spark) — CLI example
aws emr create-cluster \
  --name "ml-preprocess" \
  --release-label emr-7.1.0 \
  --applications Name=Spark \
  --instance-groups \
    InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m5.xlarge \
    InstanceGroupType=CORE,InstanceCount=4,InstanceType=m5.2xlarge \
  --use-default-roles \
  --auto-terminate

• --auto-terminate automatically shuts down the cluster when the work finishes, preventing wasted cost.
• EMR Serverless runs Spark/Hive jobs without the burden of managing a cluster, and sits somewhere between Glue and EMR.
• EMR nodes come in three kinds. The primary node (cluster coordination), core nodes (HDFS storage + compute), and task nodes (compute only). Spot is usually attached to task nodes.

Glue vs EMR Selection Table

The Glue vs EMR choice is within exam scope. The decision criteria are (1) Operational burden: minimal management → Glue, fine-grained control → EMR. (2) Workload nature: short and sporadic → Glue, long and heavy recurring batches → EMR. (3) Framework variety: if it is Spark only, Glue is enough; if you need multiple engines, EMR.

Design Principles for Large-Scale Preprocessing

Once data exceeds a few terabytes, "how you store and read it efficiently" matters as much as "how you transform it."

1. Use columnar formats — Using Parquet/ORC instead of CSV/JSON greatly improves compression ratio and scan efficiency. You read only the columns you need and reduce I/O.
2. Partitioning — If you split partitions by a frequently filtered key, like s3://bucket/year=2026/month=06/, Spark/Athena skips unnecessary data (partition pruning).
3. File size optimization — Too many small files (the small files problem) increase overhead. Compacting them to roughly 128MB to 1GB is the usual recommendation.
4. Pushdown (predicate pushdown) — Apply filter conditions at the data source stage to reduce the amount read in the first place.

# Efficient read in Spark using partition + column pruning
df = (spark.read.parquet("s3://ml-train/clicks/")
      .filter("year = 2026 AND month = 6")   # partition pruning
      .select("user_id", "event_time", "amount"))  # column pruning

[Inefficient]  CSV · no partitions · 1 million small files
               → full scan + per-file overhead → job time and cost explode

[Efficient]    Parquet · dt/region partitions · 256MB per file
               → reads only the needed partitions and columns → scan volume is a fraction (tens of times smaller)

⚠️ Pitfall: "The data is big, so let's just spin up a big EMR cluster" is a common wrong-answer lure. If the work is sporadic and operations staff is small, running a big EMR cluster around the clock is inefficient in both cost and management. In that case Glue or EMR Serverless is more suitable.

⚠️ Pitfall: It is easy to think the finer the partitions the better, but if every partition produces small files, things actually get slower. Design the partition count and the file size together.

The Minimum You Need to Know About the Spark Execution Model

Whether it is Glue or EMR, what runs inside is Spark. To read performance-problem scenarios you need at least a basic picture.

Driver  ──split work──▶  Executor × N
   · builds the job plan          · does the actual computation per partition
   · collects results             · caches intermediate results in memory
        ▲                                   │
        └──────── shuffle: redistributing data across nodes ────┘

• Partition: The chunk of data Spark processes in parallel. The number of partitions is the degree of parallelism.
• Shuffle: When a redistribution by key is needed, as with groupBy or join, data moves between nodes. It is the most expensive operation.
• Data skew: If data piles up on a particular key, only the executor handling that partition runs for a long time and the whole job slows down.
• Lazy evaluation: Transformations are not executed right away; they are optimized and executed all at once when an action (write, count) is called.

"The job is slow → make the instances bigger" is not always the answer. If the shuffle is the bottleneck, changing the join strategy (broadcasting the small table) or spreading out the skewed key is more effective.

What to Know About Running Glue Jobs

• Job bookmark: Remembers data already processed so that the next run processes only what newly arrived. The basic mechanism for incremental ETL.
• Worker type: Choose between standard workers and memory-optimized workers. If shuffles and joins are heavy, the one with more memory is safer.
• Retries and timeouts: Set the retry count and maximum execution time in the job definition to prevent endless runs and cost blowups.
• Python Shell job: For light scripts that do not need Spark, running them as a Python Shell job is much cheaper.

Connecting to SageMaker

Preprocessing results ultimately flow into SageMaker training. AWS provides several paths that connect transformation to training.

• SageMaker Processing — A SageMaker-native feature that runs preprocessing in scikit-learn/Spark containers. Often used for pre- and post-training processing.
• Glue → S3 → SageMaker Training — The most common pattern, where the Parquet built by Glue sits in S3 and the training job reads it.
• SageMaker Data Wrangler — Design the transformation flow in a GUI and export it as a Processing job or a pipeline.

SageMaker Processing and Glue seem to overlap in role, but they have a different character. Glue is strong at ETL across the data lake (producing cleansed data shared by multiple consumers), while SageMaker Processing integrates naturally with preprocessing tied to a specific ML job (feature generation for this model only).

Tomorrow we look at the training data pipeline that automates these transformation jobs with Step Functions and SageMaker Pipelines instead of running them by hand each time.

📖 Glossary

• ETL / ELT : Extract-Transform-Load / Extract-Load-Transform. The latter is the flexible variant of the data lake era.
• DPU (Data Processing Unit) : Glue's unit of processing capacity. Equivalent to about 4 vCPU + 16GB of memory.
• DynamicFrame : Glue's data structure that handles rows without discarding them even when the schema is uneven.
• Glue Data Catalog : A central metadata store containing schemas, locations, and partitions. Shared by Athena, EMR, and Redshift Spectrum.
• EMR : A managed big data cluster that runs Hadoop, Spark, Hive, and Presto directly on EC2.
• EMR Serverless : An EMR option that runs only Spark/Hive jobs without launching a cluster.
• Spot Instance : EC2 that is much cheaper in exchange for being interruptible. Used for interruption-tolerant batch computation.
• Partitioning : Splitting data paths by a frequently filtered key. Reduces scan volume.
• Shuffle : A Spark operation that redistributes data across nodes by key. The most expensive and the one most likely to become a bottleneck.
• Job bookmark : A Glue feature that remembers the point already processed so the next run handles only new data.