Day 4 - Synthetics·RUM·Evidently: Three Lenses Measuring User Experience

So far all observability pointed inward—CPU, memory, error rates, cold starts. Yet every metric green while users complain "site is slow." Specific CDN edge slow, JavaScript breaks in certain browser, DNS hiccups in region, backend fast but frontend render slow—possible. System metrics show "server healthy" but not "user actually experiences quality." Today: bridging that gap.

Three tools measuring user experience deep-dive. Synthetics (external simulation traffic active measurement), RUM (real browser passive collection), Evidently (compare two variants A/B test). Not just "tools exist," but why both active and passive monitoring need each other, how RUM safely collects from anonymous browsers, what statistical foundation backs A/B testing. In DOP-C02, distinguishing these three ("external availability," "real user LCP," "feature A/B") is standard fare; tool matching alone wins points.

Active vs Passive — Two Monitoring Paradigms

Fundamentally different approaches to user experience measurement. Active monitoring (synthetic): system generates fake traffic directly measuring. Passive monitoring (real user): observe actual user traffic.

These aren't competitors—complementary. Synthetics is proactive—bots run even at 3am with no users, finding outages first. Passive monitoring becomes representative—real users' diverse devices, networks, regions show experience robots can't mimic. Need both: Synthetics endlessly confirm "site up now," RUM observe "what real users actually experience."

💡 Related Theory: Active/passive split is decades-old in network ops. Active: ICMP ping, traceroute, iperf send probes directly. Passive: NetFlow, sFlow, packet capture observe real traffic. Active advantage: controlled repeatability (same measurement at intervals even without traffic). Passive advantage: reality representativeness (patterns real load can't fake). SRE's black-box (user view) vs white-box (internals) overlap here. Synthetics is black-box (outside like user), system metrics white-box (inside like components), RUM is real black-box (actual user). Good observability uses all three.

CloudWatch Synthetics — Robots Mimicking Users

Synthetics runs Canary scripts periodically checking site/API externally. Canary runs on Lambda, uses Puppeteer (Node.js) or Selenium (Python) spinning real browsers acting like users.

import urllib3
from aws_synthetics.common import synthetics_logger as logger
 
def heart():
    http = urllib3.PoolManager()
    r = http.request("GET", "https://api.example.com/health")
    if r.status != 200:
        raise Exception(f"Status {r.status}")
    if b'"status":"ok"' not in r.data:
        raise Exception("Body mismatch")
    logger.info(f"OK: {r.data}")
 
def handler(event, context):
    return heart()

Canary types are exam points:

• Heartbeat: single URL responds
• API: REST endpoint call/validate
• Broken Link Checker: crawl site links finding dead ones
• Visual Monitoring: screenshot pixel-diff against baseline detecting UI change/break
• GUI Workflow Builder: multi-step simulation (login→cart→checkout)

Canary failure notifies via SNS to Slack/PagerDuty.

🔍 Deeper: Synthetics' Visual Monitoring uses computer vision image diffing. Compare baseline screenshot and current pixel-by-pixel, not for exact identity but allowance threshold (e.g., fail if >5% difference) and ignore regions (ads, timestamps change every time). Same lineage as frontend test visual regression—Percy, Chromatic, Playwright screenshot comparison. CSS one-liner breaks layout but HTTP returns 200—functional test misses but Visual Monitoring catches. "Response normal but screen broke" is exactly Visual Monitoring's domain.

📚 Case Study: E-commerce company monitored API health (Heartbeat) only, missed checkout failure. Homepage, product page, login all responded, but cart→checkout multi-step payment widget failed to load. Single-URL Heartbeat couldn't catch this. Fix: GUI Workflow Canary simulating "login→add product→proceed checkout" end-to-end 5-minute intervals. Lesson: critical user journeys (CUJ) need multi-step Canary, not single endpoints. Users experience the full journey, not the sum of parts.

Synthetics Cost — Economics of Frequency

Canary runs cost Lambda per execution. 1-minute interval = ~43,200/month/Canary. Dozens of Canaries become non-trivial.

Principle: scale frequency to importance. Critical path (checkout, login) get 1-minute tight watching. General pages 5/15-minute. Not all 1-minute is cost-justified.

🔍 Deeper: Canary frequency is trade-off between MTTD (mean time to detect) and cost. 1-minute = detect failure within 1 minute but 5x cost vs 5-minute. Add alarms' "M out of N" (Day 1)—don't fire on one miss but "3 of 3 failures"—MTTD becomes period × debounce count (1-min + 3 evaluation = max 3 min). Set SLO MTTD goal first, then reverse-engineer frequency and debounce. "Always 1-minute" misses the point that MTTD is cost/benefit trade-off.

CloudWatch RUM — Real User Browser

RUM (Real User Monitoring) opposite direction: real user browser telemetry. JS snippet in page; when user loads, browser sends load time, JS errors, HTTP errors, clickstream to AWS RUM service.

<script>
(function(n,i,v,r,s,c,x,z){
  x=window.AwsRumClient={q:[],n:n,i:i,v:v,r:r,c:c};
  // ...AWS RUM client code...
})(
  'app-monitor-id','identity-pool-id','1.0.0','ap-northeast-2',
  'https://client.rum.us-east-1.amazonaws.com/1.x/cwr.js',
  {sessionSampleRate:1, guestRoleArn:'arn:aws:iam::...:role/RUM-Unauth',
   identityPoolId:'ap-northeast-2:abc'}
);
</script>

Two core things. First, auth. Anonymous web visitor browser—how does it get permission to send AWS data? Cognito Identity Pool issues guest credentials. Unauthenticated user gets limited-permission (RUM events only) temporary credentials. Second, sampling. sessionSampleRate controls collection rate—1.0=100%, 0.1=10%. High traffic dials down for cost control.

RUM collects Web Vitals—Google-defined user-experienced performance standard: LCP (Largest Contentful Paint, main content loading), INP/FID (interactivity responsiveness), CLS (layout stability). Flows to CloudWatch Metrics for alarming ("LCP p99 > 3s → alert").

💡 Related Theory: RUM using Cognito Identity Pool's unauthenticated identity solves "grant minimum permission to untrusted client safely." Browser JS is inherently untrusted (code visible, modifiable to anyone)—never embed long-lived IAM keys. Identity Pool via STS issues short-lived, extremely limited (RUM PutRumEvents only) temporary credentials. OAuth token delegation, mobile apps reaching AWS without backend keys—same pattern. Core: long-lived secrets never to client, short-lived minimum-permission tokens via delegation. Standard for all client trust-boundary problems.

⚠️ Trap: Setting sessionSampleRate to 1.0 (100%) on high-traffic site explodes costs. RUM charges per event collected; millions-daily-visitor 100% collection is unsustainable. Too low (1%) rare-browser/region bugs disappear statistically. Standard compromise: segment sampling—normal traffic 10%, post-deploy or VIP 100%, biasing value-weighted. Not "all-or-1%" extremes but intentional sample design.

CloudWatch Evidently — Testing Hypotheses Statistically

Synthetics and RUM "measure now." Evidently: "does change improve experience"—testing hypotheses not guessing. New checkout UI truly raise conversion or just prettier? Data, not hunches.

Evidently structure: Project → Feature → Variation → Launch / Experiment:

• Feature: on/off or multi-variation feature flag
• Launch: gradual rollout (10% → 50% → 100%)—canary deploy concept
• Experiment: split traffic between two variations statistically comparing metrics

aws evidently create-project --name myapp
aws evidently create-feature \
  --project myapp --name new-checkout \
  --variations 'control={boolValue=false},treatment={boolValue=true}'
aws evidently start-launch \
  --project myapp --launch '...' \
  --groups 'control=10,treatment=10' \
  --metric-monitors '...'

Launch (gradual, risk-managed) vs Experiment (statistics, hypothesis test) differ in goal. Launch "safely roll out." Experiment "prove which is better."

🔍 Deeper: Evidently's Experiment uses Bayesian statistics internally. Traditional A/B tests use frequentist p-value + fixed sample size (can't peek mid-test—peeking problem). Bayesian updates probability "treatment beats control" continuously as data arrives, answering "treatment winning now 95%" mid-test. Observability's Anomaly Detection (Day 1) uses statistical models too—AWS wraps statistical decision-making as managed services freeing ops from being statisticians. A/B testing pitfalls (sample size, exposure bias, multiple comparisons) still need careful design though—tool doesn't auto-solve it all.

⚠️ 2024 deprecation: Evidently is deprecated; new adoption discouraged, integrating into AWS AppConfig Feature Flags direction. Yet still exam-tested, so concept (Launch=gradual, Experiment=statistical comparison) and purpose (A/B testing) must be known. Production path trending: AppConfig Feature Flag + separate analytics.

Three Tools Distinction — Test's Core

DOP tests scenario-match these three. Matching table clear:

Plus 2023-24 diagnosis tools: Internet Monitor (specific city/ISP reaching service, BGP/route change visualization) and Network Monitor (VPC internal network health/reachability) worth knowing.

🎯 Scenario: "Global service but specific country users say 'slow.' Our server metrics (CPU, latency) all green." Diagnose how? Server green means application-external problem (network path/CDN/DNS likely). (1) RUM region-split data reveals which region/device/browser slow (real measurement). (2) Internet Monitor shows that region's ISP·path status and service reachability problems. (3) Deploy Synthetics Canary that region (multi-region execution) to reproduce externally. Core: "server green but users slow" means system metrics aren't the answer—user/network-view tools (RUM/Synthetics/Internet Monitor).

Summary

Today's four takeaways. First, active (Synthetics) and passive (RUM) monitoring complement—Synthetics black-box probe proactively checking without traffic, RUM real black-box (actual users). Second, Synthetics' five Canary types (Heartbeat/API/Broken Link/Visual/GUI Workflow) simulate users; frequency trades MTTD and cost on critical paths. Third, RUM uses Cognito Identity Pool unauthenticated credentials granting untrusted browser minimum-permission safely, collects Web Vitals, samples for cost control. Fourth, Evidently A/B tests (Bayesian stats) verify hypotheses but deprecated toward AppConfig Feature Flag.

Next: Week 10 synthesis. How these tools weave—detecting (Synthetics, alarms) → notifying (Composite, SNS) → diagnosing (Insights, X-Ray) timeline, tying all week's concepts into real incidents.

📌 Today's Summary

There are four key takeaways today. First, active (Synthetics) and passive (RUM) monitoring are complementary — Synthetics is a black-box probe that proactively checks availability without traffic, while RUM is the true black box that looks at real users' real experience. Second, the five Synthetics Canary types (Heartbeat/API/Broken Link/Visual/GUI Workflow) imitate users, and the interval is a trade-off between MTTD and cost, so only critical paths are checked densely. Third, RUM collects Web Vitals by delegating short-lived, least-privilege credentials to untrusted browsers through Cognito Identity Pool anonymous access, and controls cost through sampling. Fourth, Evidently validates hypotheses through A/B experiments (Bayesian statistics), but with its deprecation announced the direction is consolidation into AppConfig Feature Flags, and Internet Monitor/Network Monitor are additional diagnostic tools added in 2023-24.