Day 1 - AWS Infrastructure Through an Operator's Eyes: Regions, AZs, and the Boundary of Responsibility

📚 Case study: The December 7, 2021 us-east-1 outage. AWS's internal network auto-scaling system ran away, paralyzing the us-east-1 control plane (EC2 API, STS, IAM console, Cognito, Connect) for nearly 9 hours. The interesting part is that even though EC2 instances in other regions kept running fine, Netflix, Disney+, Slack, Robinhood, and Ring (Amazon's own IoT subsidiary) all went down together. There were three reasons. First, the STS global endpoint (sts.amazonaws.com) was actually an alias that called us-east-1 (regional endpoints can now be enforced, but the SDK default is still global). Second, the IAM console and Route 53 control planes live in us-east-1, so operators in other regions also experienced "can't log into the console." Third, many companies' CI/CD pipelines were calling us-east-1 ECR without caching. AWS official postmortem. Operator lesson: even a service labeled global may be tied to a specific region internally. If you assume "it's global, so it must be safe" without knowing this, you will have an incident.

🔍 Deeper dive: IAM, Route 53 (public hosted zones), CloudFront, WAF (the CloudFront-attached portion), Organizations, and the AWS account itself are displayed as "Global," but internally their control planes live in us-east-1, with edge or per-region data planes caching and replicating that data. In other words, if us-east-1 goes down, write operations are blocked — creating new IAM users, changing policies, modifying Route 53 records. Meanwhile, data plane operations survive: calling EC2 in other regions with already-issued IAM credentials, DNS responses for already-deployed Route 53 records, cache responses from CloudFront edges. This is called "control plane / data plane separation," and it's a core architectural principle that Werner Vogels (AWS CTO) emphasizes in his re:Invent keynote every year. In SOA-C02 scenarios, questions like "how do you keep operating during a us-east-1 outage?" frequently test this separation.

💡 Related theory: Cross-region isolation is the largest-scale implementation of the bulkhead pattern (Michael Nygard, Release It!, 2007). Like ship design that adds bulkheads so one flooded compartment doesn't sink the whole vessel, explicit barriers prevent one region's control plane failure from propagating to another. Chapter 22 of the Google SRE Book (Beyer et al., 2016), "Addressing Cascading Failures," covers the same principle: "define your failure domains clearly and design fault tolerance on top of them." AWS's region isolation is the result of elevating that failure domain to the geographic level.

Availability Zones (AZs): The Operator's First Isolation Tool

An AZ is a physically and logically independent group of one or more data centers within the same region. AZs are connected by low-latency fiber with RTT under 1-2ms, enabling synchronous replication, yet they are separated far enough (usually a few to tens of kilometers, outside the impact radius of tornadoes and floods) that a power, cooling, or network failure in one AZ doesn't spread to another. Even data centers within the same AZ use separate power and cooling, and each DC has N+1 redundancy with diesel generators, UPS, and dual power grids.

Three operator principles for handling AZs:

1. Distribute every stateful resource across at least 2 AZs: RDS Multi-AZ, Aurora clusters (auto-distributed across 3 AZs), ElastiCache replication groups, MSK brokers, OpenSearch domains
2. Distribute every stateless workload evenly across AZs via ASG: specify 3 or more in the ASG's AvailabilityZones parameter + enable AZRebalance
3. Keep operations itself untied from any single AZ: bastions, jump hosts, NAT GWs, ALB target groups, ECS services — all placed per AZ

[Region: ap-northeast-2 (Seoul)]
   │
   ├─ AZ-a (apne2-az1)  ┐
   ├─ AZ-b (apne2-az2)  ├─ Synchronous replication possible (RTT 1-2ms)
   ├─ AZ-c (apne2-az3)  │  Multi-AZ RDS / ASG distribution / NLB Cross-Zone
   └─ AZ-d (apne2-az4)  ┘  Aurora cluster automatic 6-way replication

⚠️ Pitfall: A NAT Gateway is an AZ-scoped resource. If the NAT GW in AZ-a dies, the private subnets in AZ-a lose internet access. Creating a NAT GW in only one AZ to save money becomes the single point of failure operators encounter most often. One NAT GW per AZ is the standard, so that when one AZ fails, the private subnets in other AZs stay healthy. For the same reason, back in the NAT Instance (EC2-based NAT) era, you had to build self-healing yourself with an ASG. The NAT GW is the managed service where AWS does that for you. Another pitfall: if you don't create VPC Endpoints, even S3 and DynamoDB traffic goes through the NAT GW and incurs a $0.045/GB data processing charge. Gateway Endpoints (S3/DynamoDB) are free, so enabling them is standard practice.

🔍 Deeper dive: Cross-AZ traffic is automatically encrypted on the AWS backbone (since 2022, all new EC2 instance types are Nitro-based with hardware-accelerated AEAD encryption). The same applies within a single AZ. So even if an operator doesn't apply TLS at the application level, no plaintext flows on the wire. However, compliance audits (PCI-DSS, HIPAA) often separately require application-layer encryption, so applying both is the standard.

💡 Related theory: AZ isolation design implements the distributed systems principle of failure domain isolation. The same concept appears in Google's "Borg" paper (Verma et al., 2015 EuroSys) and Microsoft's "Service Fabric" whitepaper. The core idea: "clearly define the smallest unit across which failures propagate, then design replication and failover on top of it." AWS's AZ elevates that smallest unit to the physical data center level. Through the lens of the CAP theorem (Brewer 2000, Gilbert & Lynch 2002), Multi-AZ within a region chooses CP (Consistency + Partition tolerance), while cross-region asynchronous replication is the AP trade-off. Through PACELC (Abadi 2012), AWS Multi-AZ is closer to a system that "when there is no partition, gives up some consistency for latency (PA/EL)."

📚 Case study: The August 23, 2019 Tokyo region AZ failure. In one AZ of ap-northeast-1, a bug in the cooling system control software caused some servers to overheat, affecting EC2 and EBS. At the time, workloads distributed multi-AZ via ASG survived, while everything pinned to a single AZ went down. Companies running RDS Single-AZ lost their databases too and took hours to recover. After this incident, AWS began announcing "AZ-level maintenance events" more proactively via PHD.

ZoneId vs ZoneName: A Trap Only Operators Know

Calling describe-availability-zones returns two identifiers.

+-------------------+-----------+
| ap-northeast-2a   | apne2-az1 |
| ap-northeast-2b   | apne2-az2 |
| ap-northeast-2c   | apne2-az3 |
| ap-northeast-2d   | apne2-az4 |
+-------------------+-----------+

ZoneName is shuffled per account. That means your account's ap-northeast-2a and a partner's account's ap-northeast-2a may be different physical AZs. In contrast, ZoneId (apne2-az1) is identical across all accounts. Why shuffle? If AWS told every account "create in zone a first," all traffic would pile into a single zone. Shuffling is a clever trick to force traffic distribution.

Why does this matter to operators? When you connect to another account via VPC Peering or PrivateLink and want to "co-locate in the same AZ to save cross-AZ cost and latency," you must match on ZoneId. If you match only by ZoneName, you may actually land in different physical AZs, adding 1ms of latency and incurring data transfer charges. SOA-C02 scenario questions like "how do you reduce cross-AZ costs between two accounts?" are testing this ZoneId matching.

Edge and Special Infrastructure: 4 Things Operators Must Know

On the operator exam, edge infrastructure usually appears as "how do you reduce latency in this scenario?" or "where should you apply the WAF rules?"

📚 Case study: During the 2020 COVID traffic surge, Netflix barely touched ISP backbones thanks to Open Connect Appliances (OCA) installed directly inside ISP networks. AWS uses a similar concept: CloudFront edges peer directly with ISP networks (in Seoul, direct connections to KT, LG U+, and SKB). This is why simply enabling CloudFront lets user ↔ origin traffic bypass the ISP backbone. Meanwhile, Origin Shield (GA in 2020) funnels all edge cache misses through a single Regional Edge Cache, cutting origin load by another level.

🔍 Deeper dive: CloudFront Functions and Lambda@Edge both run at the edge, but in different places. CloudFront Functions execute directly at 600+ edge PoPs and respond with sub-100μs cold starts, but with constraints: 2MB memory, 1ms execution time, JavaScript ES5 only. Lambda@Edge runs at the 13 Regional Edge Caches and can use full Node.js/Python runtimes, but cold starts are in the tens of milliseconds. On SOA-C02: "URL rewrites, header manipulation?" → CloudFront Functions; "image resizing, token validation?" → Lambda@Edge — answer this way and you'll almost always be right.

The Shared Responsibility Model: Redrawn from the Operator's Viewpoint

AWS = Security OF the Cloud / Customer = Security IN the Cloud. Everyone memorizes this one line, but if you enumerate what an operator actually has to do daily in "my area of responsibility," your feel for exam scenarios changes.

What operators actually do daily, weekly, and quarterly:

📚 Case study: The July 2019 Capital One breach. Card application data for 106 million people was exfiltrated. The cause was not AWS infrastructure but an SSRF vulnerability in the customer's WAF + IAM credential exposure via EC2 metadata v1 (IMDSv1) + overly broad IAM role permissions. The attacker (former AWS employee Paige Thompson) used SSRF to reach http://169.254.169.254/latest/meta-data/iam/security-credentials/, stole temporary credentials, and pulled 30TB of data from more than 700 S3 buckets. Nothing was wrong in AWS's area of responsibility; the incident occurred in the customer's area (WAF configuration, IAM permission scope, use of IMDSv1) — that is the key point. DOJ indictment. Right after this incident (November 2019), AWS released IMDSv2 (session-token based), and enforcing HttpTokens=required on new EC2 instances became the operator standard. Capital One paid a $190 million fine to the OCC.

🔍 Deeper dive: Enforce IMDSv2 with the SSM Document AWS-EnforceEC2InstanceMetadataServiceV2 or Launch Template MetadataOptions.HttpTokens=required. Stronger still: use an SCP forcing the aws:RequestTag/MetadataV2=required condition on RunInstances, or detect non-compliant instances with the Config rule ec2-imdsv2-check. And if you set hop limit 1 alongside, metadata traffic cannot escape outside containers (it can't cross the docker0 interface of Docker's default bridge network). This is a triple-defense pattern that stops operators from accidentally launching instances with IMDSv1.

⚠️ Pitfall: The trap of "it's a managed service, so AWS is responsible for security too." Even with Lambda or RDS, data classification, IAM permissions, encryption key policies, network access control, and backup policies are always the customer's responsibility. AWS patches the RDS OS and DB engine, but choosing the Maintenance Window in which to apply the patch, planning traffic handling during that window, and verifying that client connection pools reconnect after failover to the standby are the operator's job. Also frequently missed: RDS snapshots have a default retention of 1-35 days, so keeping them longer requires a separate export.

💡 Related theory: The shared responsibility model maps precisely onto the cloud service model classification (IaaS / PaaS / SaaS) of NIST SP 800-145. Of the five functions of the NIST CSF (Cybersecurity Framework) — Identify, Protect, Detect, Respond, Recover — Identify and Protect remain almost entirely in the customer's domain. ISO 27017 (cloud security) and ISO 27018 (cloud privacy) also codify responsibility areas on top of this model. In Korea, KISA's Cloud Security Guide (2023 revision) follows the same split, and the responsibility matrix must be submitted during K-ISMS-P certification audits.

The Operator's First Tool: AWS Health Dashboard

How do you learn fastest that us-east-1 is down? Searching Twitter? Downdetector? A colleague's Slack? All fast, but low reliability. The operator's standard answer is the AWS Health Dashboard and the AWS Health API.

Health splits into two layers.

• Service Health Dashboard (health.aws.amazon.com/health/status): Region/service-level announcements shown identically to all users. Usually delayed 1-30 minutes. Updated when AWS judges there is "customer impact"
• Personal Health Dashboard (PHD) / AWS Health API: Filters to only events affecting resources in your account. Scheduled EC2 instance restarts, EBS volume health issues, RDS maintenance notices, and Certificate Manager expirations show up here. Generally appears before the SHD

Operators receive PHD events via EventBridge and wire them into automation.

🔍 Deeper dive: The AWS Health API is hosted in only two places — us-east-1 and us-west-2 — and fails over automatically. If us-east-1 goes down, the SDK automatically retries against the us-west-2 endpoint (built into SDKs since 2023). To receive Health events in EventBridge without gaps, create aws.health source rules in both us-east-1 and us-west-2. For organization-level visibility, enable AWS Health Organizational View, which requires the management account's service-linked role (AWSServiceRoleForHealth_Organizations). Sending Health events to ServiceNow or PagerDuty follows the standard EventBridge → SNS → external webhook pattern.

Operator Scenario Patterns That Appear Frequently on the Exam

SOA-C02 is less about abstract concepts and more about the situation → which tool, and how do you respond? pattern. That's why this entire book repeats the same thought process.

[Symptom]                       [First check]         [Second check]
─────────────────────────────────────────────────────────────────
EC2 not responding           → CloudWatch metric  → EC2 status check
RDS write failure            → RDS events        → CloudWatch RDS logs
S3 403                       → CloudTrail        → Bucket Policy / IAM
Lambda timeout               → CloudWatch Logs   → X-Ray
Cost spike                   → Cost Explorer     → CloudTrail (writes)
Failure after deployment     → Deploy history    → Config Timeline
Suspected security incident  → GuardDuty         → CloudTrail Lake
Auto scaling not triggering  → ASG activity      → Scaling policy + metric

This table is the basic skeleton of the operator's thought process. From Week 2 onward, as we dig deep into each service, this pattern gains flesh. When the exam asks "which tool should you check first," this table is the answer.

Wrapping Up

Today we drew two pictures. First, AWS infrastructure is built on a 3-layer isolation of Region > AZ > Edge, and operators decide every day which resources go in which layer. Second, security and operational responsibility split along a clear boundary — AWS from the hardware to the hypervisor, and the customer above that — and the operator's daily work happens in that "above" area. As the Capital One incident showed, even when AWS is doing its job well, if we misconfigure SGs, IAM, or IMDSv1, the data of 106 million people leaks. That is the reality of cloud operations.

Tomorrow we go into the part of that upper area that breaks most often and causes the most incidents: IAM. Just as the root cause of Capital One was ultimately IAM permission scope + IMDSv1, this is the number one place where incidents happen while operators aren't looking.