Day 5 - Week 5 Synthesis: Data Transformation 2 — Engines, Orchestration, Optimization Integration Review

📌 Key Takeaways

• Choose a transformation engine by scale and operational burden: Lambda (immediate processing of small files, Firehose transformation, glue code; failures isolated to a DLQ), Glue (serverless ETL), EMR Serverless (big data without a cluster), EMR on EC2 (scale, control, ecosystem).
• Orchestration splits into Step Functions (AWS-native serverless), MWAA (Airflow assets, multiple environments), and Glue Workflows (Glue only).
• The essence of optimization is reducing the data to read. Check in this order: columnar format → splittable compression → low-cardinality partitions → 128 MB to 1 GB files.
• Limits decide the choice. Lambda is 15 minutes per invocation and 10 GB of memory; Step Functions Express is 5 minutes and cannot use wait-for-job-completion (.sync).
• The standard for EMR cost is spot Task nodes, data in S3 (EMRFS), and Transient clusters launched only while working.

Week 5 Map at a Glance

This week covered "how to transform data, how to coordinate the flow, and how to make it cheap and fast." If Glue up to Week 4 was the foundation of serverless ETL, Week 5 is the larger toolbox around it: EMR for large-scale processing, Lambda for lightweight event processing, orchestration to weave flows together, and performance and cost optimization that runs through everything. Today we tie these four axes back into a single decision map.

[Transformation engine]    [Orchestration]           [Optimization layer]
 Lambda  : light, event      Step Functions: AWS-native  Format: Parquet/ORC
 Glue    : serverless ETL    MWAA (Airflow): complex/multi Compression: Snappy/Zstd
 EMR     : large, control    Glue Workflows: Glue only   Partitions: date, etc.
 EMR S.  : serverless big data                           File size: 128MB~1GB

These three columns are not independent. In a real pipeline the three axes mesh as "the orchestrator calls transformation engines in order and stores the results in an optimized format and partitions."

1) Choosing a Transformation Engine: The Scale of Scale and Control

The core of Days 1-2 is the "trade-off between job scale and operational burden."

What you must remember about EMR: Core nodes (storage + compute) are on-demand, Task nodes (compute only) are spot. Keep permanent data in S3 (EMRFS) and save cost by launching a Transient cluster only when working. HDFS is temporary storage.

🎯 Scenario: Design one company's pipeline. (1) IoT devices keep uploading small JSON files to S3 → Lambda cleans them the moment they arrive. (2) Large-scale aggregation of accumulated data every early morning → EMR Transient (spot Task) or Glue. (3) Store results for analytics → Parquet + date partitions. (4) Coordinate the whole flow → Step Functions. The key point is that each tool is used only in its area of strength.

2) Orchestration: Three Tools That Weave the Flow

The conclusion of Day 3 is organized by keyword matching.

Pitfall reconfirmed: "many steps = MWAA" is not the rule. If everything is AWS services and you want low operations, Step Functions is right. The clues for MWAA are "Airflow assets," "multiple environments," and "complex Python logic."

The reason all three tools use a DAG is that an acyclic graph can be topologically sorted, so they can execute in "dependency-satisfied order" and parallelize independent tasks. Orchestration is ultimately "the job of defining a DAG and providing execution guarantees (retries, idempotency, visibility)," and the three tools provide those execution guarantees in a serverless way, a standard Airflow way, and a Glue-built-in way respectively.

3) Optimization: Minimize the Data Read

Memorize Day 4's four weapons in priority order.

1. Format: CSV/JSON → Parquet/ORC (column pruning + statistics skipping)
2. Compression: splittable Snappy/Zstd. No huge single Gzip text file (kills parallelism)
3. Partitioning: low-cardinality columns you filter on often (date, Region). No high cardinality (explosion of small files)
4. File size: compact to 128 MB to 1 GB. Tens of thousands of small files is overhead hell

One-line summary of formats: analytic queries = Parquet/ORC (columnar), schema evolution and fast writes = Avro (row-based).

⚠️ Pitfall: Optimization questions usually ask for several remedies at once. The answer to an "Athena cost spike" scenario is often not the single "switch to Parquet" but the combination of format conversion + partitioning + compaction. When the answer choices ask for "the single most effective action," format (columnar) or partitioning is usually first, but "apply all of them" type questions also appear.

Integrated Decision Flow

The order of thinking when you meet a question:

Q1. What is the processing scale?
    Small/event → Lambda
    Medium to large, serverless → Glue / EMR Serverless
    Very large / diverse ecosystem / cost control → EMR on EC2

Q2. How is the flow coordinated?
    AWS-native serverless → Step Functions
    Airflow assets / multiple environments → MWAA
    Pure Glue → Glue Workflows

Q3. How is storage optimized?
    Parquet + Snappy + date partitions + 128MB~1GB

Limits and Execution Model Comparison: Where Numbers Decide the Choice

Wrong answers on engine-selection questions are usually eliminated by limits. Let us put the four engines side by side on the same items.

• Bundle the remaining Lambda limits too. The /tmp ephemeral storage is 512 MB to 10,240 MB, and the payload is 6 MB synchronous and 1 MB asynchronous (the asynchronous limit was 256 KB for a long time, so exam materials may show 256 KB). CPU is allocated in proportion to memory, so a function short on CPU gets faster by raising its memory.
• EMR Serverless pre-initialized capacity warms workers in advance so jobs start within seconds, but idle cost arises accordingly. If the condition is "intermittent batch, no idle cost," do not turn it on.
• "A single-file transformation that takes 30 minutes" is not for Lambda even if the data is small. The reason for elimination is not size but the time limit.

Step Functions Standard vs Express: Where They Diverge for ETL

On Day 3 we distinguished them only by execution time (1 year vs 5 minutes). For ETL orchestration there are more important differences.

• The default for ETL orchestration is Standard. A Glue Job or EMR Step takes minutes to hours, and integration patterns that wait until completion, like glue:startJobRun.sync, cannot be used with Express.
• Standard can be entrusted with non-idempotent tasks. Since the same state is not executed twice (except where you wrote Retry), it suits tasks like creating an EMR cluster. Express can run one extra time, so use it only for idempotent tasks.
• The billing intuition is reversed. Standard gets more expensive with more steps, Express with longer and more frequent runs. Running hundreds of short events per second through Standard makes the number of state transitions explode.
• The workflow type cannot be changed after the state machine is created.

Who Retries a Failure

Pipeline-failure questions ask "which layer retries, and where does the failure remain?"

⚠️ Pitfall: If you stack retries on several layers, they multiply. If you set 3 Retry in Step Functions together with 3 retries on the Glue Job itself, the Job can run up to 16 times for a single failure. Put retries on only one layer (usually the orchestrator), and design output so that re-running gives the same result, by overwriting per partition rather than appending.

One More Point to Make About EMR

• The premise behind "spot Task nodes are safe." Task nodes do not run the HDFS DataNode, so data is not lost even if they are reclaimed. But if the YARN application master that directs the job was running on that node, the job itself fails. From EMR 6.x the default is that the application master can run on either Core or Task nodes, so if the spot share is large, configure it with node labels to place it only on Core (or on-demand) nodes.
• The Primary node can be configured as three nodes for high availability. With a single-node configuration, the Primary is a single point of failure. This is why you must not put it on spot.
• Managed Scaling: you just set the minimum and maximum capacity, and EMR watches metrics and grows and shrinks the Core and Task nodes. It works only for YARN-based applications (Spark, Hive, Flink, and so on) and does not apply to Presto or HBase. The Primary node is not a target of adjustment.
• Instance Fleets have mixing of purchase options and instance types as a basic feature, which helps secure spot capacity. Instance Groups have one type and one purchase option per group.

Billing Structure in One Table

Scenario Practice: Filter Out the Wrong Answers First

Next week we move to the side that stores and serves data: the data warehouse and catalog area. It covers where and how the transformed data settles.

📖 Glossary

• Transient cluster: A one-time EMR cluster that is launched when a job starts and terminates automatically when it finishes.
• EMRFS: The layer that lets EMR read and write S3 like a file system. It lets you keep data permanently outside the cluster.
• Core / Task nodes: Core does HDFS storage and compute together, and Task does compute only. Task nodes can disappear with the data still remaining, so they suit spot.
• Cold start: The phenomenon where the first invocation is slow because Lambda creates a new execution environment.
• State transition: In Step Functions, one step finishing and moving on to the next. It is the billing unit of Standard workflows.
• DAG: An acyclic graph that draws tasks as nodes and dependencies as arrows. It is the basis on which an orchestrator decides execution order.
• Column pruning: An optimization in columnar formats that picks and reads only the columns the query needs.
• Partition pruning: An optimization that reads only the partition folders matching the condition and does not open the rest.
• Compaction: The task of merging many small files into larger files of appropriate size.
• Idempotency: The property that running the same task multiple times gives the same result as running it once. It is required for retries to be safe.

📝 Practice Questions

Click a choice to reveal the answer and explanation.

Question 1

Which transformation engine satisfies all of the following requirements? "Process tens of TB, use Hive and HBase together, control cost aggressively with Spot Instances, and finely tune the cluster."

• AAmazon EMR on EC2
• BAWS Lambda
• CAWS Glue
• DAmazon Athena

Question 2

You want to orchestrate "S3 file arrives → Glue transformation → validation → load into Redshift if it passes, notify on failure" in a serverless way with branching and retries. Which orchestrator is the most suitable?

• AAmazon MWAA
• BA single Lambda function
• CAmazon Kinesis
• DAWS Step Functions

Question 3

Which statement about EMR cost optimization is correct?

• AStore persistent data in HDFS on the Core nodes and keep the cluster running 24 hours a day
• BApply Spot to Task nodes, keep persistent data in S3 (EMRFS), and launch a transient cluster only when running jobs
• CConfigure the Primary node with Spot to reduce cost
• DFix all nodes as On-Demand

Question 4

Athena scan costs have spiked. The data is stored as a single Gzip CSV, and queries look only at the last few days and specific columns. Which of the following actions would NOT help reduce cost?

• AConvert to a columnar format such as Parquet
• BPartition by year/month/day
• CPartition finely by a high-cardinality column such as user_id and split the files further
• DCompact small files to 128 MB to 1 GB and use splittable Snappy compression

Question 5

Which of the following pairs of a tool and its typical use is INCORRECT?

• ALambda — lightweight event-driven transformation that immediately cleanses small files arriving in S3
• BMWAA — complex workflows that reuse existing Airflow DAGs and coordinate multi-cloud environments
• CGlue Workflows — general-purpose orchestration that coordinates a variety of non-Glue services with complex branching
• DParquet — a columnar format that reduces the amount scanned through column pruning in analytical queries