"Keeping data pipelines running" is harder than "making them run in the first place." Today we cover monitoring pipelines via CloudWatch, tracking Glue, EMR, Kinesis services, and tracing latency.
Operational visibility typically consists of metrics, logs, and traces. In AWS these map to CloudWatch Metrics, CloudWatch Logs, and AWS X-Ray respectively. Data pipeline monitoring almost always starts with CloudWatch; most AWS services auto-publish operational metrics to CloudWatch namespaces without configuration.