Day 5 - Week 10 Synthesis: Monitoring as One Story

Over one week we examined CloudWatch, X-Ray, CloudTrail, EventBridge separately. Yet these four are actually one question answered from four angles: "What's happening in the running system, how do we know, and how do we respond?" This final day doesn't just review individual services; it shows how these four interlock into one observability picture, then validates that understanding through comprehensive scenario questions.

Four Services on One Coordinate

The most common exam mistake is memorizing "name → function." Instead, two axes create coordinates revealing why each tool sits where it does.

First axis: what do we observe — numbers (metrics)? text (logs)? request path (traces)? action records (audit)? Second axis: when it acts — passive observation only, or real-time response?

From this coordinate: CloudWatch answers "what's slow" but X-Ray answers "where." CloudTrail tracks responsibility, not performance. EventBridge isn't an observer but a responder — others "know," EventBridge "acts." The most powerful pattern combines: CloudTrail detects root login, EventBridge responds with Lambda calling SNS → operator notification.

💡 Related theory: Modern monitoring redefines as "observability" — control theory term: "from external outputs, how much can we infer internal state?" Observability's three pillars are metrics, logs, traces — exactly CloudWatch (metrics + logs) and X-Ray (traces). CloudTrail adds fourth: audit — security/compliance observability. Key insight: "monitoring watches expected problems (alarms pre-set); observability lets us forensically explore unexpected problems post-hoc." CloudTrail enables post-incident forensics. Observability requires all three pillars; monitoring may use only one.

Week 10 Core Traps: Organized

Exams test nuance. Here's one week's traps consolidated.

⚠️ Trap: Top-3 mistake sources. ① EC2 memory, disk space aren't basic metrics — hypervisor can't peer inside OS. ② CloudTrail Data Events default OFF — S3 GetObject, Lambda Invoke need explicit enable. ③ ALB doesn't support X-Ray — tracing starts from API Gateway. These three account for half of Week 10 wrong answers.

Integrated Architecture: Signal Flow

[Application]
     |
     +-- API call ────→ [CloudTrail] ──→ [EventBridge] ──→ [Lambda] ──→ Security Response
     |                                          (root login pattern match)
     |
     +-- Distributed request ───→ [X-Ray] (Trace ID propagation, service map)
     |
     +-- Performance metrics ────→ [CloudWatch Metrics] ──→ [Alarm] ──→ [SNS / Auto Scaling]
     |                                    ↑
     |                              [EMF / Metric Filter]
     |                                    ↑
     +-- Logs ──────────→ [CloudWatch Logs] ──→ [Logs Insights / Subscription Filter]

Critical: arrow direction. CloudTrail, X-Ray, CloudWatch receive signals from application. EventBridge and Alarm loop back to respond automatically. Observation and response cycle is monitoring's essence.

🔍 Going deeper: Often missed: "logs → metrics → alarms" transformation. Logs (text) alone can't gate alarms — alarms need metrics (numbers). Metric Filter and EMF bridge the gap. Without this transform, "ERROR in logs → alert" isn't direct; it must become "ERROR count (metric) > threshold." CloudWatch's all auto-responses reduce to "metric → alarm → action," all other signals (logs, events) convert to join this path.

Week 10 Abbreviations

---

📝 Week 10 Comprehensive Practice

📝 Practice Questions

Click a choice to reveal the answer and explanation.

Question 1

API Gateway → Lambda → DynamoDB → external payment API chain, intermittent timeout. To identify which stage delays:

• AView Lambda Duration/Errors metrics; compare avg/p99 per invocation
• BX-Ray service map and traces showing per-stage duration
• CCloudTrail API call timestamps to estimate delay gaps
• DVPC Flow Logs packet round-trip time to back-trace

Question 2

EC2 monitoring needs: memory usage, disk space used (%), and DiskReadOps. What's required?

• AAll three are basic hypervisor-observable
• BMemory/disk space need Agent; DiskReadOps is basic
• CAll require OS-level Agent measurement
• DDetailed Monitoring enables all three auto-collect

Question 3

Compliance audit: S3 object downloads (GetObject) — who, when — must retain 7 years. Appropriate strategy:

• AManagement Events suffice, auto 7-year S3 retention
• BData Events enable (extra cost), Trail → S3 or CloudTrail Lake
• CCloudWatch Logs auto-records GetObject
• DX-Ray traces download

Question 4

Microservice: "search all traces for specific userId in X-Ray console," plus "attach order JSON body for debug (not searched)." Each needs what?

• ABoth in Metadata; indexing enables search + payload
• BuserId→ Annotation (searchable); JSON → Metadata (big, unsearched)
• CBoth Annotation (50/trace limit handles both)
• DuserId in Segment name; JSON in Annotation

Question 5

Lambda business metric to CloudWatch, minimize cost/latency. EC2/ECS X-Ray traces not visible. Separate solutions:

• ALambda PutMetricData sync, EC2 Active Tracing toggle
• BLambda EMF stdout output; EC2/ECS X-Ray daemon sidecar confirm
• CBoth CloudWatch Agent install
• DLambda X-Ray; EC2 EMF

Question 6

Root account console login → instant Slack alert. Standard pattern:

• AIAM policy block root login
• BCloudTrail logs ConsoleLogin → EventBridge Rule root filter → Lambda/SNS → Slack
• CCloudWatch metric alarm on login count
• DAWS Config Rule evaluates root

Question 7

Traffic low/high/weekend variance makes single CPU threshold fail — "80% oversensitive early morning, undersensitive noon." Solution without manual threshold swap:

• AMultiple static thresholds, EventBridge schedule swaps hourly
• BCloudWatch Anomaly Detection (ML seasonality band)
• CMetric Filter log pattern → time-stratified metrics
• DComposite Alarm AND/OR multiple alarms

Question 8

"Error count > 100 alarm" triggered when traffic 10x, alerting constantly despite normal error rate. More robust alarm:

• ARaise threshold to 1000 for 10x headroom
• BMetric Math: error rate (errors/total × 100); alarm on ratio not count
• CAnomaly Detection on error count
• DLengthen eval period, more datapoints to smooth

Question 9

Stream CloudWatch metrics to Datadog real-time AND application logs to OpenSearch. Each tool/method:

• ABoth via Metric Stream to Firehose, Datadog and OpenSearch targets
• BMetrics → Metric Stream; Logs → Subscription Filter
• CBoth Subscription Filter streams metrics and logs together
• DBoth GetMetricData/FilterLogEvents API polling

Question 10

ECS Fargate: one task memory-starved, restarts repeatedly; host average shows normal. Container-level visibility:

• AEC2 basic metrics sufficient
• BContainer Insights; cluster/service/task/container-level granularity
• CCloudWatch Agent on host
• DX-Ray

Question 11

Schedule: "exactly 9 AM daily," scaling to thousands of future schedules. Appropriate expression + tool:

• Arate(1 day) rule ensures 9 AM
• Bcron(0 9 * * ? *); large scale → EventBridge Scheduler
• CCloudWatch alarm on 9 AM metric
• DSQS 24-hour delay queue periodic cycle

Question 12

Post-deploy: API returns 200, but payment button on screen broken, unclickable. Metrics show all normal. Detect "technically works, user-broken" automatically:

• ACloudWatch basic metric alarm
• BSynthetics canary scripted flow (login→payment) + Visual monitoring (screenshot pixel diff for breaks)
• CX-Ray sampling
• DCloudTrail Data Events

Wrapping Up

Week 10's four services form one cycle: "observe, respond." CloudWatch observes what (metrics), X-Ray observes where (causal), CloudTrail observes who (responsibility), EventBridge responds. Traps mostly ask fine details on this unified pipeline — EC2 memory boundary, Data Events default, Annotation indexing, Lambda toggle vs. EC2 daemon. Understanding each tool's coordinate ("what/when") rather than memorizing alone lets you tackle novel scenarios: find its coordinate, deduce the answer.