Day 4 - Synthetics Canary, X-Ray Sampling, ServiceLens: User-Centric Monitoring Design

Incidents always follow the saying: "users find out first." In reality, there are many cases where internal metrics look fine but user experience is broken. The reverse is also true—metrics are red while users keep using the service without any problems. If an operator only looks at system internals, it's hard to handle both situations correctly. Today's topics—Synthetics Canary, RUM, X-Ray, and ServiceLens—are tools for setting up monitoring from the "user perspective" rather than from "inside the system."

Synthetics Canary: History and Design Philosophy of Synthetic Monitoring

"Synthetic" monitoring is a method where a virtual user (bot) executes a predefined scenario without real traffic to measure availability and performance. The concept itself originated in the early 2000s as an "internet measurement" business from companies like Gomez and Keynote. With the arrival of Puppeteer (Google, 2017)—which programmatically controls browsers—realistic synthetic monitoring with browser automation became possible.

CloudWatch Synthetics Canary internally runs Lambda functions that execute Chromium (Puppeteer-based) or Python scripts. The canary script written by the user runs in the Lambda execution environment, stores execution results (screenshots, HAR files, logs) to S3, and publishes success/failure metrics to the CloudWatchSynthetics namespace.

[EventBridge Scheduler] → (by rate or cron)
      │
      ▼
[Lambda Execution Environment]
  canary script executes
  Chromium/Python access URL
  validate responses
      │
      ├─→ [S3] screenshots, HAR files, execution logs
      ├─→ [CloudWatch Logs] detailed execution logs
      └─→ [CloudWatch Metrics] SuccessPercent, Duration, Failed

Canary Runtimes and Blueprints

Canary runtimes come in two families: Node.js (Puppeteer-based) and Python. In practice, the selection criterion is primarily the team's language familiarity.

AWS provides blueprints for quick starts.

The Visual Monitoring Blueprint compares a baseline screenshot with the current screenshot, and considers it a failure if the pixel difference percentage exceeds a threshold. This is useful for automatically detecting layout breaks after UI deployments.

// Heartbeat Canary basic structure
const synthetics = require('Synthetics');
const log = require('SyntheticsLogger');
 
const heartbeat = async function () {
  const page = await synthetics.getPage();
 
  // 1. Load page
  const response = await page.goto(
    'https://api.example.com/health',
    { waitUntil: 'networkidle0', timeout: 30000 }
  );
 
  // 2. Validate HTTP status code
  if (response.status() !== 200) {
    throw new Error(`Expected 200, got ${response.status()}`);
  }
 
  // 3. Validate response body
  const body = await response.json();
  if (body.status !== 'healthy') {
    throw new Error(`Unhealthy: ${JSON.stringify(body)}`);
  }
 
  // 4. Save screenshot
  await synthetics.takeScreenshot('health-check', 'pass');
  log.info('Health check passed');
};
 
exports.handler = async () => {
  return await synthetics.executeStep('heartbeat', heartbeat);
};

Comparing Synthetics and RUM: Complementary Perspectives

RUM's Core Web Vitals are user experience metrics proposed by Google in 2020. The three key metrics are LCP (Largest Contentful Paint), FID (First Input Delay), and CLS (Cumulative Layout Shift). They also impact Google search rankings, directly affecting SEO.

📚 Case Study: According to 2019 Amazon research, a 100ms increase in page loading decreases revenue by 1%. Walmart reported a 2% improvement in conversion rate when improving page speed by 1 second. RUM is the data source that connects these business metrics with technical metrics. While Synthetics only measures "the experience of an average bot," RUM measures the actual experience of "a mobile user on LTE from Seoul."

X-Ray: Internal Structure of Distributed Tracing and Sampling Algorithm

AWS X-Ray traces the flow of requests through multiple services in distributed systems. The foundational concept comes from Google's Dapper paper (Sigelman et al., 2010). Dapper first proposed a hierarchical request tracking structure using trace_id and span_id, which later became the foundation for Zipkin (Twitter), Jaeger (Uber), and OpenTelemetry. X-Ray implements the same concept adapted for the AWS environment.

Trace: The complete path of a single request from creation to completion. Propagated between services via the X-Amzn-Trace-Id HTTP header.

Segment: The processing period of one service (Lambda, EC2, ECS). Includes start time, end time, error status, and HTTP metadata.

Subsegment: Detailed operations within a Segment. SDK calls like DynamoDB queries, S3 uploads, and external API calls are automatically captured as Subsegments.

Trace (entire request)
  └─ Segment: API Gateway
      └─ Segment: Lambda order-service
          ├─ Subsegment: DynamoDB GetItem (3ms)
          ├─ Subsegment: SQS SendMessage (8ms)
          └─ Subsegment: HTTPS call to payment-api (245ms)  ← bottleneck!
              └─ Segment: Lambda payment-service
                  └─ Subsegment: RDS Query (230ms)  ← root cause

When this tree structure is visualized as a Service Map, you immediately understand which service causes delays.

X-Ray Sampling Rule: Balancing Cost and Precision

If X-Ray tracked every request, costs would explode with high-volume traffic. At $5 per million traces, a 1000 RPS service with 100% sampling would generate 43.2 million traces per day = $6,480/month X-Ray cost alone. Impractical.

X-Ray Sampling operates on a Reservoir + Fixed Rate model. Reservoir is a "bucket" that guarantees "at least N traces per second." Fixed Rate is a percentage applied to additional requests after the Reservoir fills.

Default sampling rule:
  Reservoir = 1 (at least 1 per second guaranteed)
  Fixed Rate = 5% (5% of remainder)

Example (100 RPS per second):
  - First 1 request: guaranteed to trace
  - Remaining 99 × 5% = 4.95 ≈ 5 requests
  → Total roughly 6 traces per second = 6% effective sampling rate

Custom Sampling Rules can specify different sampling rates by Priority, service name, HTTP method, and URL path.

# Payment API 100%, health check 0%
aws xray create-sampling-rule \
  --sampling-rule '{
    "RuleName": "payment-api-full",
    "Priority": 1,
    "FixedRate": 1.0,
    "ReservoirSize": 50,
    "ServiceName": "payment-service",
    "ServiceType": "*",
    "Host": "*",
    "HTTPMethod": "POST",
    "URLPath": "/api/v*/payments/*",
    "ResourceARN": "*",
    "Version": 1
  }'
 
aws xray create-sampling-rule \
  --sampling-rule '{
    "RuleName": "health-check-exclude",
    "Priority": 2,
    "FixedRate": 0.0,
    "ReservoirSize": 0,
    "ServiceName": "*",
    "ServiceType": "*",
    "Host": "*",
    "HTTPMethod": "GET",
    "URLPath": "/health",
    "ResourceARN": "*",
    "Version": 1
  }'

Priority 1 matches first. The /health endpoint gets Priority 2 rule with 0% sampling (no tracing at all). The default rule has Priority 10000, lower than any user-defined rule.

🔍 Deeper Dive: The Reservoir size is managed by the X-Ray central sampling service. When one service runs in multiple instances, if each instance independently collects "1 per second," over-collection occurs system-wide. The X-Ray SDK communicates with the central sampling service to distribute the Reservoir. If there are 10 instances, each gets 1/10 of the Reservoir. When network isolation prevents reaching the central service, "fallback sampling (fixed 5%)" applies.

X-Ray Activation Methods and Differences

When Lambda is set to Mode=Active, Lambda's own execution time is auto-traced. However, to trace DynamoDB calls inside Lambda as Subsegments, SDK integration is required.

const AWSXRay = require('aws-xray-sdk-core');
const AWS = AWSXRay.captureAWS(require('aws-sdk'));  // Auto-capture all AWS SDK calls
 
const ddb = new AWS.DynamoDB.DocumentClient();
// Now ddb.get(), ddb.put() etc. are auto-traced as X-Ray Subsegments
 
// Also trace external HTTP calls
const https = AWSXRay.captureHTTPs(require('https'));
 
// Custom Subsegment
const segment = AWSXRay.getSegment();
const sub = segment.addNewSubsegment('custom-calculation');
try {
  doComplexCalculation();
  sub.close();
} catch (e) {
  sub.addError(e);
  sub.close();
  throw e;
}

ServiceLens: Integrated View of Three Signals

ServiceLens overlays CloudWatch Metrics and Logs on top of the X-Ray Service Map in a unified operations console. It's not an independent service but a view within the CloudWatch console.

Operations enabled by ServiceLens:

• Click Service Map node → see p50/p95/p99 latency and error rate for that service
• Node's "View traces" → actual trace list passing through that service
• Click Trace → all Segments/Subsegments of the trace + timing waterfall
• "View logs" → jump instantly to CloudWatch Logs for that Lambda execution by trace_id
• Alarm overlay → visualize which service has ALARM status

This "Trace ID → Logs jump" feature is the most powerful. You can find a trace of a specific 5xx error and open the Lambda function's execution logs in under a second. Otherwise you'd manually search CloudWatch Logs by request_id.

💡 Related Theory: ServiceLens implements the concept of Correlated Telemetry. Rather than viewing each signal (Metrics, Logs, Traces) independently, you look at the same request/event through different signals and cross-reference them. OpenTelemetry (CNCF, 2019) was born to standardize this correlation. By inserting log_group_name and request_id as correlation attributes into Traces, you can connect data from different storage systems as a single event. AWS provides this correlation through native integration of X-Ray and CloudWatch Logs.

Container Insights: ECS/EKS-Specific Monitoring

Container Insights is auto-monitoring exclusively for container environments. EKS uses CloudWatch Agent + Fluent Bit as a DaemonSet, and ECS uses awslogs driver or Fluent Bit sidecars.

# EKS: Enable Container Insights
aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name amazon-cloudwatch-observability \
  --addon-version v1.7.0-eksbuild.1

Collected metrics:

• Cluster level: Total CPU/memory utilization, Pod count
• Node level: Per-node CPU/memory/network
• Pod level: Per-pod CPU/memory utilization, restart count
• Container level: Per-container CPU/memory

Auto-generated Performance Dashboards enable hierarchical drill-down from cluster → node → pod → container.

⚠️ Pitfall: Container Insights incurs additional costs. Billing is based on the number of metrics collected and log volume. Depending on EKS cluster size, tens to hundreds of dollars may be added monthly. It's recommended to review whether you need all container metrics and optimize settings to collect only necessary namespaces.

Production Architecture: Synthetics + X-Ray + ServiceLens Combination

[Synthetics Canary]         [Real Users]
 1-minute heartbeat          RUM JS snippet
 API canary 5-minute          │
       │                     │
       ▼                     ▼
 [CloudWatch Metrics]    [CloudWatch RUM]
 SuccessPercent<90% → ALARM  LCP/CLS/JS errors

       User request
          │
     API Gateway (X-Ray Active)
          │
     Lambda order-service (X-Ray Active)
     ├─ DynamoDB (Subsegment auto)
     └─ SQS (Subsegment auto)
          │
     Lambda payment-service (X-Ray Active)
     └─ RDS (Subsegment auto)
          │
     [X-Ray Service Map → ServiceLens]
     Find error node → Trace → Logs jump instantly

Wrapping Up

Synthetics validates "the service is alive" 24 hours a day. RUM measures "what real users actually experience in terms of performance." X-Ray finds "which service is making it slow" through distributed tracing. ServiceLens integrates these three signals to "open this error trace's Lambda logs" in one click. Each of the four tools has its own domain, and they're most powerful when used together.