2:43am, on-call engineer's phone rings three times. 15 PagerDuty alerts. EC2 CPU High, RDS Connections High, ALB 5xx High, ElastiCache CPU High, SQS Queue Depth High... all from same incident. Root cause was one, engineer overwhelmed by 15 alerts, doesn't know where to start. Alert fatigue textbook form.
CloudWatch Alarm is AWS operations' most basic automation trigger. But simple "exceed threshold → alert" design exhausts overnight on-call. Today explores alarm state machine, M of N evaluation algorithm, TreatMissingData, Composite Alarm, and Anomaly Detection internals.
CloudWatch Alarm has exactly three states