Day 3 - CloudWatch Agent: Internal Operations, StatsD/collectd Integration, procstat Plugin

When you open the Monitoring tab in the EC2 console, you see CPU, network I/O, and disk I/O. But the metrics operators most want to see—memory usage and disk free space—are missing. The reason is simple: these metrics are only knowable from inside the guest OS, and the AWS hypervisor cannot see inside the guest OS. This is where the boundary of the shared responsibility model becomes apparent. CloudWatch Agent is software that runs inside that boundary—inside the guest OS—and exports metrics and logs to CloudWatch.

Why Agent Is Needed: Hypervisor Boundary and Measurability

AWS EC2 runs on the Nitro hypervisor. Nitro can see hardware-level metrics like CPU cycles, network packets, and block I/O. But how the guest OS is using memory, how full the file system is for each mount point, and which processes are consuming CPU—only the guest OS kernel knows this.

This is why there's no mem_used_percent in EC2's basic metrics. This constraint isn't unique to EC2. GCP Compute Engine equally requires Ops Agent (formerly Stackdriver Agent) to be installed for guest OS internal metrics. Azure VMs also require Azure Monitor Agent.

💡 Related Theory: Monitoring approaches broadly split into agent-based and agentless. Agentless means observing from the outside through SNMP, IPMI, or hypervisor APIs. Agent-based means installing software inside the measurement target to collect richer data. In cloud environments, both are typically mixed: platform level (CPU, network) is agentless, and OS level (memory, disk, process) is agent-based. Understanding this hybrid structure lets you immediately answer "which metrics come automatically and which require an Agent?"

CloudWatch Agent's Internal Architecture

CloudWatch Agent is a single binary written in Go. Internally, it performs three roles simultaneously.

Metric Collector: Reads OS metrics from procfs(/proc), sysfs(/sys), Windows WMI/Performance Counter. On Linux, mem_used_percent is parsed from /proc/meminfo. disk/used_percent is calculated using the statfs() system call on each mount point's block usage.

Log Collector: Monitors specified file paths in a tail -f fashion, and when new lines are added, sends them via CloudWatch Logs API (PutLogEvents). When multiline patterns are specified, multiple lines like Java stack traces are bundled into a single log event.

Protocol Server: Listens for StatsD (UDP 8125) and collectd (UDP 25826) protocols and relays custom metrics sent by applications to CloudWatch.

[EC2 Guest OS]
  │
  ├─ /proc/meminfo, /proc/diskstats → [Metric Collector] ─┐
  ├─ /var/log/app/*.log → [Log Collector] ────────────────┤
  └─ StatsD :8125, collectd :25826 → [Protocol Server] ───┘
                                                        │
                                              [CloudWatch API]
                                              PutMetricData / PutLogEvents

Configuration File Structure: Complete Example

Understanding the JSON structure generated by the configuration wizard (amazon-cloudwatch-agent-config-wizard) makes manual editing and automation easier.

{
  "agent": {
    "metrics_collection_interval": 60,
    "run_as_user": "cwagent",
    "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log",
    "debug": false
  },
  "metrics": {
    "namespace": "MyApp/EC2",
    "append_dimensions": {
      "InstanceId": "${aws:InstanceId}",
      "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
      "ImageId": "${aws:ImageId}"
    },
    "aggregation_dimensions": [["AutoScalingGroupName"]],
    "metrics_collected": {
      "mem": {
        "measurement": ["mem_used_percent", "mem_available_percent", "mem_cached"],
        "metrics_collection_interval": 60
      },
      "disk": {
        "resources": ["/", "/var", "/home"],
        "measurement": ["used_percent", "inodes_free", "disk_free"],
        "ignore_file_system_types": ["sysfs", "devtmpfs", "tmpfs"],
        "metrics_collection_interval": 60
      },
      "cpu": {
        "totalcpu": true,
        "per_cpu": false,
        "measurement": ["usage_user", "usage_system", "usage_iowait", "usage_steal"],
        "metrics_collection_interval": 60
      },
      "procstat": [
        {
          "exe": "nginx",
          "measurement": ["cpu_usage", "memory_rss", "memory_vms",
                          "num_threads", "read_bytes", "write_bytes"],
          "metrics_collection_interval": 60
        },
        {
          "exe": "java",
          "measurement": ["cpu_usage", "memory_rss", "num_fds"],
          "metrics_collection_interval": 60
        }
      ],
      "statsd": {
        "service_address": ":8125",
        "metrics_collection_interval": 10,
        "metrics_aggregation_interval": 60
      },
      "collectd": {
        "service_address": "udp://127.0.0.1:25826",
        "name_prefix": "collectd_"
      }
    }
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/myapp/app.log",
            "log_group_name": "/prod/myapp/app",
            "log_stream_name": "{instance_id}",
            "timezone": "UTC",
            "timestamp_format": "%Y-%m-%dT%H:%M:%S.%f",
            "multi_line_start_pattern": "^\\d{4}-\\d{2}-\\d{2}",
            "retention_in_days": 90
          },
          {
            "file_path": "/var/log/nginx/error.log",
            "log_group_name": "/prod/nginx/error",
            "log_stream_name": "{instance_id}",
            "retention_in_days": 30
          }
        ]
      },
      "windows_events": {
        "collect_list": [
          {
            "event_name": "System",
            "event_levels": ["ERROR", "CRITICAL"],
            "log_group_name": "/windows/system-events",
            "log_stream_name": "{instance_id}"
          }
        ]
      }
    }
  }
}

aggregation_dimensions is an important advanced feature. When set to [["AutoScalingGroupName"]], metrics from all instances in an ASG are aggregated by ASG name Dimension. In addition to per-instance metrics, a "whole ASG" metric is emitted separately. This is useful for seeing "fleet average memory" in an autoscaling environment.

procstat Plugin: Process-Level Monitoring

procstat is a plugin that tracks specific processes. The target process is specified by exe (executable name), pid_file (PID file path), or pattern (regex).

"procstat": [
  {
    "exe": "nginx",
    "measurement": ["cpu_usage", "memory_rss", "num_threads", "read_bytes", "write_bytes"]
  },
  {
    "pid_file": "/var/run/myapp.pid",
    "measurement": ["cpu_usage", "memory_rss", "num_fds", "involuntary_context_switches"]
  },
  {
    "pattern": "python.*worker",
    "measurement": ["cpu_usage", "memory_rss"]
  }
]

memory_rss is Resident Set Size (actual physical memory footprint), while memory_vms is Virtual Memory Size (virtual address space size). RSS is more meaningful for memory leak detection. When num_fds (file descriptor count) approaches the OS limit, a Too many open files error occurs. Tracking this proactively with procstat is preventative monitoring.

📚 Case Study: A SaaS company (anonymous) in 2020 had API servers go unresponsive every Saturday at dawn. EC2 CPU and memory were normal. The cause was nginx's num_fds reaching 65535 (Linux default ulimit). If num_fds had been monitored with procstat and an alarm set at 80% threshold, Saturday dawn incidents could have been prevented. After this incident, the team added procstat num_fds + num_threads monitoring as a standard on all servers.

StatsD and collectd: Application Custom Metrics Integration

StatsD is a lightweight metrics aggregation protocol developed by Etsy in 2011 and released as open source. It sends metrics via UDP packets with minimal performance impact on applications.

# StatsD metric format
order.count:1|c           # Counter (cumulative)
order.latency:45|ms       # Timer (ms)
cart.size:3|g             # Gauge (current value)
payment.success:1|c|@0.1  # 10% sampling

CloudWatch Agent receives StatsD packets on UDP 8125 and publishes them to CloudWatch. If the aggregation interval (metrics_aggregation_interval) is 60 seconds, it sums the counters received over 60 seconds and publishes them as a Sum statistic.

# Python application StatsD usage example
import statsd
c = statsd.StatsClient('localhost', 8125, prefix='payment')
 
def process_payment(amount):
    with c.timer('latency'):  # Measure processing time
        result = do_payment(amount)
    if result.success:
        c.incr('success')       # Success counter
        c.gauge('amount', amount)  # Latest payment amount
    else:
        c.incr('failure')
    return result

collectd is a system statistics collection daemon developed since 2005. When integrating on-premises servers previously monitored with collectd into CloudWatch, the Agent's collectd support is useful. During migration, you can keep existing collectd configuration intact while sending data to CloudWatch.

🔍 Deeper Dive: Embedded Metrics Format (EMF) is a different approach from StatsD/collectd. EMF inserts metric data in a special JSON format into CloudWatch Logs, and CloudWatch automatically parses and converts it to metrics. This is useful in environments like Lambda or ECS Tasks where running an agent is difficult. The aws-embedded-metrics library abstracts this. EMF preserves both logs and metrics simultaneously, so you retain "the log context at the moment the metric occurred"—a significant advantage. In SOA-C02, EMF appears as the standard pattern for custom metrics in serverless environments.

Comparison with Other Platforms

SSM Parameter Store Integration and State Manager Automation

The core of operations automation is deploying and maintaining identical Agent configuration across 100 EC2 instances. The standard pattern is three steps.

# Step 1: Save configuration to SSM Parameter Store
aws ssm put-parameter \
  --name "/cloudwatch/agent/prod-web" \
  --type String \
  --tier Advanced \
  --value file://amazon-cloudwatch-agent.json \
  --description "Web tier CloudWatch Agent configuration"
 
# Step 2: Initial installation (Run Command)
aws ssm send-command \
  --document-name "AWS-ConfigureAWSPackage" \
  --parameters 'action=Install,name=AmazonCloudWatchAgent,version=latest' \
  --targets "Key=tag:Role,Values=web-prod" \
  --comment "Install CloudWatch Agent on web-prod"
 
# Step 3: Apply configuration (Run Command)
aws ssm send-command \
  --document-name "AmazonCloudWatch-ManageAgent" \
  --parameters '{
    "action": ["configure"],
    "mode": ["ec2"],
    "optionalConfigurationSource": ["ssm"],
    "optionalConfigurationLocation": ["/cloudwatch/agent/prod-web"],
    "optionalRestart": ["yes"]
  }' \
  --targets "Key=tag:Role,Values=web-prod"
 
# Step 4: State Manager Association for automatic drift correction
aws ssm create-association \
  --name "AmazonCloudWatch-ManageAgent" \
  --association-name "CWAgent-WebProd-Maintain" \
  --targets "Key=tag:Role,Values=web-prod" \
  --parameters '{
    "action": ["configure"],
    "mode": ["ec2"],
    "optionalConfigurationSource": ["ssm"],
    "optionalConfigurationLocation": ["/cloudwatch/agent/prod-web"],
    "optionalRestart": ["yes"]
  }' \
  --schedule-expression "rate(24 hours)" \
  --compliance-severity "MEDIUM"

The core of State Manager Association is periodic forced application (desired state enforcement). Even if someone accidentally stops the Agent or changes the configuration on an instance, within 24 hours it's automatically restored to the original configuration. When a new instance starts with the Role=web-prod tag, the configuration is automatically applied at the next Association execution cycle.

💡 Related Theory: The concept State Manager implements is Infrastructure as Desired State. It's identical to the "idempotent execution" philosophy of Puppet, Chef, and Ansible. When you declare the desired state, the agent continuously aligns the actual state with it. The difference between "deploy once" and "maintain continuously" determines operational stability.

IAM Permissions and Troubleshooting

Two IAM policies are needed to run the Agent.

# Managed policies to grant to EC2 IAM Role
arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

Key permissions granted by CloudWatchAgentServerPolicy:

• cloudwatch:PutMetricData — Send metrics
• logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents — Send logs
• ec2:DescribeTags — Auto-collect Dimensions like InstanceId, ASG name
• ssm:GetParameter — Read configuration from Parameter Store

Troubleshooting order is IAM → Agent status → logs → time synchronization → network.

# Check Agent status
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status
 
# Agent logs (90% of error causes are here)
sudo tail -f /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log
 
# Time synchronization status (rejected metrics if time is wrong)
chronyc tracking
timedatectl show
 
# IAM permissions test (run directly on instance)
aws cloudwatch put-metric-data \
  --namespace "Test/Agent" \
  --metric-name "TestMetric" \
  --value 1 \
  --region ap-northeast-2

⚠️ Gotcha: When CloudWatch Agent publishes memory metrics, they go to the CWAgent namespace (or the value configured in namespace). They don't appear in the EC2 console Monitoring tab. When creating alarms, you must select the namespace as CWAgent (or your custom value), not AWS/EC2. Many operators get confused looking for mem_used_percent in the AWS/EC2 namespace and can't find it.

⚠️ Gotcha: When you set retention_in_days in the configuration file, the Agent automatically sets the Log Group's retention period. It only applies when the Log Group is first created; it doesn't change the retention of existing Log Groups. It's more stable to centralize retention period management via Lambda or Terraform.

Complete Flow to Finish with Memory Metric Alarms

# Memory exceeds 85% — 3 of 5 alarm
aws cloudwatch put-metric-alarm \
  --alarm-name "EC2-Memory-High-WebProd" \
  --metric-name mem_used_percent \
  --namespace CWAgent \
  --dimensions \
    Name=InstanceId,Value=i-0123456789abcdef0 \
    Name=AutoScalingGroupName,Value=web-prod-asg \
  --period 60 \
  --statistic Average \
  --threshold 85 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 5 \
  --datapoints-to-alarm 3 \
  --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:ap-northeast-2:123456789012:ops-critical

Memory metrics obviously don't arrive when an EC2 instance is terminated. Using notBreaching means the terminated instance's alarm stays OK instead of transitioning to ALARM. If your goal is availability monitoring (the instance must be alive), separately add a StatusCheckFailed_System alarm.

Summary

CloudWatch Agent is not a tool to install and forget. It's the core of a complete operations automation pipeline that tracks processes via procstat, integrates application custom metrics via StatsD/collectd, and prevents configuration drift with State Manager. In the exam, recurring patterns are: "Memory requires Agent", "IAM permissions come first", and "Parameter Store + State Manager is the standard for large-scale deployment".