Day 5 - Week 8 Comprehensive Review: Data Operations and Support Recap

This week covered data pipeline "operations" post-construction: monitoring, quality, troubleshooting, cost. Today we tie four topics into one operations scenario. Data operations and support is large DEA-C01 exam domain.

1. Pipeline Monitoring (Day 1)

• Observability 3 pillars: Metrics (CloudWatch Metrics) + Logs (CloudWatch Logs) + Traces (X-Ray).
• Glue: Continuous logging + --enable-metrics; watch task failures, shuffle metrics.
• EMR: YARN memory, ContainerPendingRatio, IsIdle for resource/cost judgment; save logs to S3.
• Kinesis latency: IteratorAgeMilliseconds key; throttling: ProvisionedThroughputExceeded.
• Custom metrics: EMF logs metric+context simultaneously; composite a