Day 1 - CloudWatch: Why Observability Split Into the Three Pillars of Metrics, Logs, and Traces

The most expensive moment in any system you operate is "there's an outage and I don't know why." The server didn't die — responses just got slow — but if you can't tell whether it's the DB, the network, or the application code, engineers keep restarting on guesswork while revenue bleeds out. This is exactly why the concept of Observability emerged as distinct from plain "monitoring." Monitoring is "answering questions I defined in advance (did CPU cross 80%?)," while observability is "the ability to answer, with data, even questions I never anticipated (why is only this one specific user slow?)." And that ability only appears when you can cross-reference three kinds of data: metrics, logs, and traces.

CloudWatch, which AWS launched in 2009, started as a simple metrics tool that graphed EC2 CPU utilization. But as the cloud fragmented from a single server into hundreds of microservices, Lambdas, and containers, CloudWatch kept expanding too: Metrics (2009) → Logs (2014) → richer Alarms → Logs Insights (2018) → Container/Lambda Insights → Synthetics and RUM. Rather than list CloudWatch's features, this article traces "why metrics and logs had to have different storage structures," "why high-resolution metrics are expensive," and "what distributed-systems problem a Subscription Filter solves" — pinning down the essence of what the SAA exam's Operational Excellence domain is really asking.

Why You Should Never Put Metrics and Logs in the Same Store

When you first learn CloudWatch, a question arises: "Why are metrics and logs separate? Aren't both just data accumulating over time?" In reality the two are fundamentally different data models, and that difference splits cost, query performance, and retention strategy all at once.

A metric is a time-series number. When you attach a dimension like InstanceId=i-123 to the CPUUtilization metric in the AWS/EC2 namespace, that is "one line a specific instance's CPU draws across the time axis." The key property of time-series data is that pre-aggregation is possible. CloudWatch doesn't store incoming raw datapoints as-is; it pre-computes and compresses min/max/sum/avg/count in 1-minute, 5-minute, and 1-hour buckets. That's why querying CPU trends from six months ago is fast and cheap to store. The trade-off is that "what exactly happened in this precise 1 second" is unknowable — it was already aggregated away.

A log is an unstructured (or semi-structured) text event. The single line "2026-05-29T10:23:01 ERROR user_id=4823 payment failed: gateway timeout" cannot be aggregated. Each line carries unique context, so none can be discarded, and thus logs are stored raw. Storage costs far more than metrics, and querying requires scanning the whole thing (which is why Logs Insights exists). In exchange, it can answer "what exactly happened in that 1 second."

💡 Related theory: This split is the architectural difference between a time-series database (TSDB) and a log search engine, plain and simple. TSDBs like Prometheus and InfluxDB store a compressed time series per metric-name-plus-label combination and shrink old data via downsampling. Log engines like Elasticsearch and Loki make text searchable via an inverted index or label-based chunks. CloudWatch put both engines inside one console, but the internal storage structures are completely different — which is why even the APIs are split: PutMetricData for metrics, PutLogEvents for logs.

🔍 Going deeper: CloudWatch metric retention automatically downsamples by resolution — 1-minute-resolution data is kept 15 days, 5-minute for 63 days, and 1-hour for 455 days (15 months). In other words, when you look at data from six months ago, only values automatically rolled up into 1-hour buckets remain. This isn't a bug; it's intentional downsampling. It's plenty for long-term trend analysis, but if you want to see "the per-minute spike from that incident three months ago," you're already too late. So when compliance or post-incident analysis matters, you must separately siphon raw metrics off to S3/Kinesis via Metric Streams.

The exam-favorite trap that EC2 memory and disk utilization aren't in the default metrics also stems from this structure. CPU, network, and disk I/O are observable at the hypervisor (Nitro/Xen) level — AWS sees them directly from the virtualization layer. But memory utilization and free disk space are information inside the guest OS. The hypervisor only knows it allocated 4GB to the guest; it has no idea whether the OS inside is using 3GB or 1GB. So to obtain memory and disk metrics, you must install the CloudWatch Agent inside the guest and have the OS report them directly. This isn't a limitation of AWS — it's an essential constraint created by virtualization's isolation boundary.

⚠️ Pitfall: The answer to "I want to create an alarm on EC2 memory utilization" is always "install the CloudWatch Agent." It's easy to misdiagnose it as a missing IAM Role or a region problem, but the key is that memory simply doesn't exist in the standard metrics at all. Going further, in container environments you use Container Insights instead of the Agent, and in Lambda you use Lambda Insights — so the responsible party for "observing inside the guest" shifts per environment.

The Real Reason High-Resolution Metrics and PutMetricData Are Expensive

CloudWatch custom metrics are published via the PutMetricData API. Whereas standard metrics are 1-minute resolution, custom metrics can go up to 1-second (high-resolution). But look at the cost structure and high-resolution metrics are far pricier than standard, and a single metric is billed separately for every dimension combination. Why so expensive?

The core is cardinality explosion. Metrics are stored one-per-"metric name + dimension combination." Attach four dimensions — Service, Region, Endpoint, UserId — to a RequestLatency metric and you get 5 services × 3 regions × 20 endpoints × 100,000 users = 30 million individual time series. Each series is stored, aggregated, and billed separately. That's why the anti-pattern of "putting user ID in a dimension" balloons cost astronomically. Dimensions should only ever be low-cardinality (service, region, endpoint); per-user tracking is the domain of logs/traces, not metrics.

🔍 Going deeper: This cardinality problem is a fundamental limit of every time-series system. The incident a Prometheus operator fears most is a "high cardinality label" — if someone accidentally puts a request ID or timestamp in a label, the series count grows without bound and the TSDB's memory blows up. CloudWatch defends against this with billing — if cardinality explodes, cost explodes, so it's naturally suppressed. Which means "what do I put in a metric dimension" is a cost decision as much as an architectural one.

EMF (Embedded Metric Format) resolves this tension elegantly. Embed a metric inside a single log line using a special JSON structure, and CloudWatch receives that log and automatically extracts the metric. That is, the application doesn't call PutMetricData at all — it just writes one log — and CloudWatch pulls both a metric and a log out of it simultaneously. It's a pattern that reduces PutMetricData API call count and cost in high-volume environments while getting both an "aggregation metric" and a "detailed debugging log" from the same log line. It fits especially well with environments where stdout automatically flows to CloudWatch Logs, like Lambda.

The Alarm State Machine and the Meaning of INSUFFICIENT_DATA

A CloudWatch Alarm looks like a simple threshold comparison, but it's actually a state machine with three states: OK / ALARM / INSUFFICIENT_DATA. And properly understanding that third state, INSUFFICIENT_DATA, is the fork in the road of alarm design.

An alarm works on the logic "if M of N datapoints cross the threshold during the evaluation period, go to ALARM" (evaluation-periods and datapoints-to-alarm). But what if there are no datapoints at all? For example, if traffic is 0 so the request metric isn't reported, the alarm becomes neither OK nor ALARM but INSUFFICIENT_DATA. And here a common incident happens — you set "alarm when error rate crosses the threshold," but when the service dies completely and requests themselves drop to 0, the error-rate metric isn't reported, so the alarm never fires. The worst-case scenario: the service is fully down but monitoring is silent.

⚠️ Pitfall: You must explicitly choose a "missing data" handling policy. CloudWatch treats missing data as one of notBreaching (treat as normal), breaching (treat as anomalous), ignore (keep current state), or missing (default, INSUFFICIENT_DATA). Availability alarms should usually set missing to breaching so that "traffic stopped" also counts as an outage. Leave it at the default and a dead service slips by silently.

That alarm actions connect not only to SNS but directly to EC2 Auto Recovery, ASG Scaling, and Systems Manager also reveals the design philosophy. An alarm isn't just a tool to notify a human — it's a trigger for self-healing. Attach an Auto Recovery action to the EC2 StatusCheckFailed_System metric, and on hardware failure the instance is relocated to healthy hardware with the same ID and same IP, without any human intervention. A Composite Alarm combines multiple alarms with AND/OR to build precise conditions like "only when the DB alarm AND the cache alarm fire together," suppressing alarm storms. Anomaly Detection uses ML to learn a metric's normal band and catches "different from usual" without a fixed threshold — you use it for services whose traffic varies greatly by time of day or day of week, where a fixed threshold is useless.

📚 Case study: Many organizations create so many alarms they fall into "alarm fatigue." When hundreds of alarms fire a day, engineers start ignoring all of them, and then they miss the one that actually matters. The principle the Google SRE book emphasizes is: "Every alarm must demand immediate human action — otherwise it's not an alarm, it's noise." To realize this in CloudWatch, bundle correlated alarms with Composite Alarms and route the auto-recoverable ones to automation actions instead of SNS, reducing the number of alarms that reach a human's inbox.

The Distributed-Systems Problem a Subscription Filter Solves

CloudWatch Logs' Subscription Filter shows up on the exam often as "process ERROR logs in real time," but behind it lies an important distributed-systems design principle. When you need to ship logs somewhere in real time, the question is which to choose: "periodically poll to read new logs" or "push the instant a log arrives."

Polling is simple but has two problems. A long poll interval hurts real-timeness; a short one wastes most polls returning "no new logs." Also, the poller must manage a cursor of how far it has read, and if it dies it either misses logs or double-processes them. The Subscription Filter solves this with a push-based subscription model — register a filter pattern on a log group, and the moment a matching log event arrives CloudWatch pushes it to a destination (Lambda / Kinesis Data Streams / Firehose). No polling, no cursor management.

That the destinations split into three is also intentional. Lambda is for low-latency, low-volume processing that invokes a function immediately per event (like an instant alert on root login). Kinesis Data Streams is for high-volume logs where order must be guaranteed and multiple consumers need to read simultaneously. Firehose is for buffering logs and batch-loading them into S3/OpenSearch/Redshift. That is, you pick the destination along three axes — "real-timeness vs throughput vs load target" — and on the exam, "multiple consumers processing the same log stream simultaneously" signals Kinesis, while "collect into S3 for analysis" signals Firehose.

🔍 Going deeper: Cross-account Subscription is used when the account sending logs differs from the one receiving them. In the pattern where a central security account gathers all member accounts' logs in one place, you create a Destination on the receiving side's Kinesis and attach a policy allowing the sending accounts. This is the foundation of multi-account log centralization, and the standard design for pulling security logs into a Log Archive account in an Organizations environment.

Logs Insights, Dashboards, and External Monitoring

CloudWatch Logs Insights queries logs with a SQL-like query language. Instead of full-scanning raw logs every time, it's a schema-on-read approach that parses only the fields you need at query time. With commands like fields, filter, stats, sort, and limit, you can ad-hoc run "aggregate the last hour's ERROR logs by endpoint." You don't have to build an index in advance, but in exchange you're billed by the volume of data scanned per query, so narrowing the time range directly affects both cost and speed.

CloudWatch Dashboards composite metric and log widgets from multiple regions and accounts onto one screen. Operating a global service, you need "unified cross-region visibility," but since metrics are region-isolated by default, a cross-region, cross-account dashboard ties this together into a single view. Synthetics actively checks site availability from the outside with a headless browser (Canary) — this is active monitoring that "finds the problem ourselves before users actually experience it" — while RUM (Real User Monitoring) conversely passively collects the performance real users experience via JS embedded in real users' browsers. Synthetics is "proactive detection with synthetic traffic," RUM is "post-hoc analysis with real-usage data" — a complementary pair.

💡 Related theory: The Synthetics-vs-RUM distinction also maps to monitoring theory's "blackbox vs whitebox." Synthetics is blackbox monitoring that knocks on the system from outside like a user (it doesn't know the internals, only sees results), while CloudWatch Agent, Logs, and EMF are whitebox monitoring that reports from inside the system. Google SRE summarizes it as "blackbox tells you 'something is broken right now,' whitebox tells you 'why it's about to break.'" You need both to see symptom and cause together.

Comparing With Other Clouds and Tools

Relativizing CloudWatch makes its design choices sharper. SaaS APMs like Datadog and New Relic bundle metrics, logs, and traces into one integrated platform from the start, so correlation analysis is seamless — but you have to export data outside and cost spikes as volume grows. CloudWatch keeps data inside AWS and integrates with IAM and KMS, but you have to fill the unified-analysis experience with a separate feature like ServiceLens. The Prometheus + Grafana open-source stack is strong on metrics and lets you directly control cost, but the operational burden is heavy and logs/traces need a separate stack (Loki, Tempo). AWS, conscious of this trend, released Amazon Managed Prometheus (AMP) and Managed Grafana (AMG), and supports standard OpenTelemetry via ADOT — offering a middle ground of "managed convenience without being locked into AWS."

🔍 Going deeper: The relationship between EventBridge and the old CloudWatch Events is worth knowing too. CloudWatch Events came first, and EventBridge emerged as its superset — they share the same event bus, but EventBridge added features like third-party SaaS events, a schema registry, and multiple event buses. New designs should use EventBridge, and on the exam the answer for "event-driven automatic response" is EventBridge. Remember the division of labor — CloudWatch Alarm → SNS is threshold-based, EventBridge rule → target is event-pattern-based — and you won't get confused.

Getting Hands-On With the CLI

# Create an Alarm (CPU > 70%, ALARM if exceeded 2 consecutive 1-min periods)
aws cloudwatch put-metric-alarm --alarm-name HighCPU \
  --metric-name CPUUtilization --namespace AWS/EC2 \
  --dimensions Name=InstanceId,Value=i-1234567890abcdef0 \
  --statistic Average --period 60 --threshold 70 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 2 --datapoints-to-alarm 2 \
  --treat-missing-data breaching \
  --alarm-actions arn:aws:sns:ap-northeast-2:111:ops-topic
 
# Auto-recover on EC2 system status failure (relocated with same ID/IP)
aws cloudwatch put-metric-alarm --alarm-name AutoRecover \
  --metric-name StatusCheckFailed_System --namespace AWS/EC2 \
  --dimensions Name=InstanceId,Value=i-1234567890abcdef0 \
  --statistic Maximum --period 60 --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --evaluation-periods 2 \
  --alarm-actions arn:aws:automate:ap-northeast-2:ec2:recover
 
# Publish a custom metric (dimensions must be low-cardinality only)
aws cloudwatch put-metric-data --namespace MyApp \
  --metric-name OrderProcessed \
  --dimensions Service=checkout,Region=apne2 \
  --value 1 --unit Count
 
# Logs Insights query (narrow the time range to optimize cost and speed)
aws logs start-query --log-group-name /aws/lambda/saa-fn \
  --start-time $(date -d '-1 hour' +%s) --end-time $(date +%s) \
  --query-string 'fields @timestamp, @message
    | filter @message like /ERROR/
    | stats count(*) as errors by bin(5m)
    | sort errors desc'
 
# Push ERROR logs to Lambda in real time via a Subscription Filter
aws logs put-subscription-filter \
  --log-group-name /aws/lambda/saa-fn \
  --filter-name error-to-lambda \
  --filter-pattern "ERROR" \
  --destination-arn arn:aws:lambda:ap-northeast-2:111:function:alert-fn
 
# Composite Alarm (only when the DB and cache alarms are both in ALARM)
aws cloudwatch put-composite-alarm \
  --alarm-name DbAndCacheDown \
  --alarm-rule "ALARM(DbAlarm) AND ALARM(CacheAlarm)" \
  --alarm-actions arn:aws:sns:ap-northeast-2:111:critical

Wrapping Up

CloudWatch handles two of observability's three pillars, metrics and logs (traces are X-Ray, Day 4), and it's AWS's standard operations tool that stacks alarms, automation, and external monitoring on top. The essence compresses into five points. ① Metrics are pre-aggregated time series, so they're cheap and fast but lose "that one second," while logs are raw, so they're expensive but preserve exact context — that's why the two storage structures are separate. ② EC2 memory and disk are information inside the guest OS, so the hypervisor can't see them and you need the CloudWatch Agent. ③ Putting a high-cardinality value (user ID) in a metric dimension explodes cost, so use only low-cardinality ones, and leave fine-grained tracking to logs/traces. ④ An Alarm is an OK/ALARM/INSUFFICIENT_DATA state machine, and availability alarms must treat missing data as breaching so they don't miss a dead service. ⑤ A Subscription Filter is a poll-free push subscription, and you choose between Lambda/Kinesis/Firehose by real-timeness, throughput, and load target.

In the next article, we look at the audit tool that answers not "what is happening now" but "who did what" — CloudTrail. If CloudWatch is the tool for seeing a system's health, CloudTrail is the tool that records the actions of people and services in a tamper-proof way, and it's the starting point of security incident analysis.