Day 1 - Pipeline Monitoring: CloudWatch Metrics, Logs, and Alarms

📌 Key Takeaways

• Observability = metrics (CloudWatch Metrics) + logs (CloudWatch Logs) + traces (X-Ray). CloudWatch is the starting point for data pipeline monitoring.
• An alarm fires when Datapoints to Alarm datapoints breach the threshold within Period × Evaluation Periods. A metric that stops arriving altogether is only caught if you decide it explicitly with TreatMissingData.
• Glue sends metrics every 30 seconds and EMR every 5 minutes. For Glue you watch task failures, shuffle, and executor counts; for EMR you watch YARN memory, pending containers, and IsIdle.
• The key streaming-lag metric is GetRecords.IteratorAgeMilliseconds (statistic: Maximum), and throttling shows up in Read/WriteProvisionedThroughputExceeded.
• For custom metrics, EMF, which emits metrics from a single log line, fits Lambda and container pipelines better than PutMetricData, and alarm storms are tamed by grouping alarms into a composite alarm.

Three Pillars of Observability and CloudWatch

With data pipelines, "confirming that they keep running" is harder than "making them run in the first place." Today we look at how to observe Glue, EMR, and Kinesis with CloudWatch at the center, and how to trace delays.

• Operational visibility is usually described through three pillars: metrics, logs, and traces. In AWS these are CloudWatch Metrics, CloudWatch Logs, and AWS X-Ray respectively.
• Most AWS services publish operational metrics to their own namespaces with no extra configuration, so pipeline monitoring almost always starts in CloudWatch.

Metric Structure: Namespaces, Dimensions, Resolution, Retention

• A metric is identified by the combination of name + namespace + dimensions. If the dimension combination differs, it is a different metric even with the same name. You can attach up to 30 dimensions to a metric.
• For AWS service metrics, omitting a dimension in a query sometimes returns an aggregate, but custom metrics are not aggregated across dimensions. They can only be queried by exactly the dimension combination they were published with. This is the cause of "I specified just one dimension and no data shows up."
• There are two resolutions. Standard resolution is 1-minute granularity and high resolution is 1-second granularity; only custom metrics can be published at high resolution.
• Metrics cannot be deleted, and they expire after 15 months if no new data arrives. The older the data, the more coarsely it is rolled up.

• Setting an alarm period shorter than the metric resolution is pointless. If you put a 1-minute alarm on a metric that arrives every 5 minutes, most periods end up as "no data."
• High-resolution metrics can have high-resolution alarms with 10-second or 30-second periods, and these alarms cost more than regular alarms.

💡 Related theory: To catch fast-changing streaming lag at second-level granularity, use a high-resolution custom metric; for ordinary batch jobs, use standard resolution. High-resolution data stays at one-second granularity for only 3 hours, so for looking at "last night's trend" it ends up being 1-minute aggregation anyway.

Custom Metrics: PutMetricData, EMF, Metric Filters

When the default service metrics are not enough, the application creates its own metrics. There are three ways.

{
  "_aws": {
    "Timestamp": 1750000000000,
    "CloudWatchMetrics": [{
      "Namespace": "DataPipeline",
      "Dimensions": [["PipelineName"]],
      "Metrics": [{ "Name": "RecordsProcessed", "Unit": "Count" }]
    }]
  },
  "PipelineName": "orders-etl",
  "RecordsProcessed": 18420
}

• EMF leaves both the metric and its context (which pipeline, which batch) in a single log line. When a metric spikes, you can find the cause right in the same log.
• If you put a field whose values grow without bound, such as a request ID, into a dimension, every dimension combination creates its own metric and costs balloon. Limit dimensions to low-cardinality values such as pipeline name or environment.

How Alarms Are Evaluated

• An alarm has three states: OK, ALARM, and INSUFFICIENT_DATA. Actions run once when the state changes (Auto Scaling actions are the only exception).
• There are four values for missing-data handling, and the default is missing.

• If a job does not run at all, the failure metric is not emitted either. With the default, the alarm just moves to INSUFFICIENT_DATA with no notification. This is the classic incident of "the pipeline stopped overnight and the alarm was silent." Set breaching on a metric such as processed count, or attach an action to the INSUFFICIENT_DATA transition as well.
• A composite alarm combines the states of other alarms with AND/OR. Attaching notifications only to the composite alarm and not to the underlying alarms reduces alarm storms. A composite alarm can send SNS notifications but cannot run EC2 or Auto Scaling actions.
• Put alarms on symptoms (rising lag, rising failure rate), and look at plain resource numbers on a dashboard.

# Notify if consumer lag stays above 1 minute for 5 minutes
aws cloudwatch put-metric-alarm \
  --alarm-name orders-stream-consumer-lag \
  --namespace AWS/Kinesis \
  --metric-name GetRecords.IteratorAgeMilliseconds \
  --dimensions Name=StreamName,Value=orders \
  --statistic Maximum --period 60 \
  --evaluation-periods 5 --datapoints-to-alarm 5 \
  --threshold 60000 --comparison-operator GreaterThanThreshold \
  --treat-missing-data breaching \
  --alarm-actions arn:aws:sns:ap-northeast-2:111122223333:data-oncall

• This metric exists only when a consumer calls GetRecords. If the consumer dies, the metric stops, so you must set breaching so that "consumer stopped" is caught by the same alarm.

Glue Monitoring

When job metrics are enabled (--enable-metrics), a Glue job sends metrics to the Glue namespace every 30 seconds. The dimensions are JobName, JobRunId, and Type (count or gauge). They are recorded as custom metrics, so CloudWatch charges may apply.

• Look at the last two metrics overlaid. If the "needed" count is consistently larger than the "running" count, DPUs are insufficient; if it is far smaller, you have provisioned too much.
• If only the driver heap spikes, suspect code that pulls data into the driver (collect) or listing hundreds of thousands of small files. If only a particular executor spikes, it is skew.
• Logs: From Glue 5.0 onward, every job writes logs in real time. On 4.0 and earlier, you must turn on Continuous Logging (--enable-continuous-cloudwatch-log) for logs to flow to CloudWatch Logs before the job ends; with it off, you can see them only after the job finishes. Exam material usually asks this as "turn on continuous logging."
• The default log groups are /aws-glue/jobs/output (user stdout and stderr) and /aws-glue/jobs/error (system and Spark logs).

# Write to the driver log stream in real time from inside the job script
logger = glueContext.get_logger()
logger.info("orders partition dt=2026-06-26 loaded")
logger.error("schema mismatch: email column missing")

💡 Related theory: A Glue Job Bookmark is not a monitoring metric; it is state that makes a job skip data it has already processed, preventing duplicate processing on rerun. However, if the bytesRead metric suddenly drops close to 0 or jumps to several times its usual level, that is a clue that the bookmark has gone wrong.

EMR Monitoring

EMR sends cluster metrics to the AWS/ElasticMapReduce namespace every 5 minutes. The interval cannot be changed, and the metrics themselves carry no charge.

• IsIdle looks only at that instant, every 5 minutes. A value of 1 does not mean the cluster idled for the entire 5 minutes, so set the alarm for when it is 1 several times in a row, such as for 30 minutes.
• The feature that shuts down an idle cluster by itself is the auto-termination policy. After the specified idle time passes, EMR terminates the cluster.
• When a cluster disappears, the nodes' logs disappear with it. You must configure step logs and Spark event logs to be kept in S3 so you can debug after termination.

Kinesis Monitoring and Tracking Lag

In streaming, "how far behind are we right now" is the heart of the matter.

• Stream-level metrics arrive every minute at no charge. Shard-level metrics (enhanced monitoring) must be turned on separately with EnableEnhancedMonitoring and are billed per shard and per metric. To see "whether only a specific shard is hot," you need to turn this on.
• Put the iterator age alarm on the Maximum statistic. Average hides a lagging shard behind shards that are keeping up.
• If this value exceeds half the retention period, records are at risk of expiring and being lost. This alarm takes priority over the throttle alarm.

Finding Answers in Logs

• If you do not set a retention period on a log group, logs accumulate without expiring. Set a retention period for each job to control storage cost.
• Logs Insights queries log groups with a query language. You are billed for the amount of log data scanned, so narrow the time range when you run it.
• A subscription filter forwards logs in real time to Lambda, Kinesis, or Firehose. Use it to archive logs in S3 for the long term or to send them to OpenSearch.

# Logs Insights: error count per 5-minute bin in recent runs
fields @timestamp, @message
| filter @message like /Exception|ERROR/
| stats count(*) as errors by bin(5m)
| sort errors desc

Symptom → Cause → Action

Question Clue → Answer Lookup Table

Tomorrow we cover the problem that a pipeline can run perfectly and still produce wrong content, that is, Glue Data Quality and validation gates.

📖 Glossary

• Namespace: The container that holds metrics. AWS services use their own namespaces such as AWS/Kinesis.
• Dimension: A name-value pair attached to a metric. Even with the same metric name, different dimensions make separate metrics.
• EMF: A format where printing a JSON log in a defined structure lets CloudWatch extract metrics automatically.
• Metric filter: A setting that counts lines matching a pattern in logs and turns them into a metric.
• Datapoints to Alarm: The value that sets how many of the evaluation periods must breach the threshold for the alarm to fire.
• TreatMissingData: An alarm setting that decides whether periods with no data are treated as healthy, breaching, or ignored.
• Composite alarm: An alarm that combines the states of other alarms with AND/OR and evaluates them as one.
• Continuous logging: A feature that sends logs to CloudWatch Logs in real time even before a Glue job ends.
• Iterator age: How long ago the record a consumer just read entered the stream. The larger it is, the further behind the consumer is.
• Enhanced monitoring: A paid option that delivers Kinesis metrics per shard.