Day 2 - Multi-Region Resilience: Distributed Principles of DNS Routing, Global Replication, and Encryption Boundaries

Within a single region using multiple AZs, you survive a burning data center. But what if an entire region—say us-east-1—fails? The major us-east-1 S3 outage in 2017, the us-east-1 Kinesis/API failures in 2021—these happen rarely but really. Multi-region architecture answers the "entire region disappears" scenario. Yet the moment you cross regions, physics intrudes: the speed of light. Round-trip latency between Seoul and Virginia exceeds 200ms, making synchronous replication (possible in single-region millisecond timescales) essentially impossible across regions. So all multi-region design reduces to: "Accept asynchronous replication and decide how you'll handle the resulting data inconsistency."

Today we examine four pillars underpinning multi-region—Route 53 routing (deciding which region to send traffic to, and the DNS/health check mechanics underneath), Aurora Global / DynamoDB Global Tables / S3 CRR (replicating data across regions), and KMS Multi-Region Key (bridging encryption boundaries. Multi-region appears in DOP exams as "automatic failover on region failure," "lowest latency to global users," "decrypt replicated encrypted data across regions." The skill is reading whether each choice implies active-active or active-passive, sync or async.

How DNS Decides Routing—Inside Route 53

Route 53 is not a simple DNS server but a global traffic director combining health checks and policies. When a user queries api.example.com, Route 53 dynamically decides which IP/endpoint to return based on configured routing policy. Two mechanisms sit underneath: DNS itself (name→address translation) and health checks (periodic verification that endpoints are alive).

💡 Related Theory: Route 53 routing is fundamentally DNS-based Global Server Load Balancing (GSLB). DNS, defined in RFC 1034/1035 (1987), is a distributed hierarchical database originally for static name resolution. GSLB adds: "dynamically change responses based on client location and server health." Yet DNS routing has a fundamental limit: DNS is cached. Resolvers, the OS, and browsers cache responses for the TTL duration, so even if failover is signaled, the old answer lives on until the cache expires. This is why Latency/Failover routing's failover is not "instant." For instant region-switch, the answer is not DNS but Global Accelerator using fixed anycast IPs (compared below).

🔍 Deeper: Route 53's health checks are not simple pings but simultaneous checks from multiple global locations (checkers), judged by quorum—relying on a single checker could misinterpret temporary network glitches as failures, so at least 18% of checkers must report healthy for the check to be deemed healthy (global consensus prevents false positives). Three types exist: Endpoint checks (HTTP/HTTPS/TCP direct probe), Calculated checks (combine multiple checks with AND/OR—"database AND cache both alive for healthy"), and CloudWatch Alarm-based checks (convert metric alarms to health—for resources you can't probe directly or for composite metrics). This mix can catch "endpoint returns 200 but is actually broken."

Active-Active and Active-Passive—Two Modes of Multi-Region

Multi-region splits into two operational modes. Active-Passive: only one region (Primary) takes traffic; others (Secondary) wait and promote on failure (typical: Route 53 Failover Routing + health checks). Active-Active: both regions handle traffic simultaneously (typical: Route 53 Latency Routing + bidirectional replication). Active-Active approaches RTO=0 (both already running, just shift traffic) but must resolve data conflicts from bidirectional writes.

Active-Active (Latency Routing)
==================================================
  Users (Global)
       │
       ▼
  Route 53 Latency Routing + Health Check
       │
       ├─► Region A (ap-northeast-2)  ── Simultaneous traffic
       │     ALB → ECS/Lambda
       │     Aurora Global writer / DDB Global Table
       │
       └─► Region B (us-east-1)        ── Simultaneous traffic
             ALB → ECS/Lambda
             Aurora Global secondary / DDB Global Table

  Cross-Region data layer:
   ├─ Aurora Global   : Async <1s (single writer)
   ├─ DDB Global Tables: Multi-master, LWW conflict resolution
   ├─ S3 CRR          : Async object replication
   └─ KMS Multi-Region Key: Encryption boundary bridging

Aurora Global Database—Single-Writer Asynchronous Global Replication

Aurora Global ties one Primary region and up to 10 Secondary regions. The key insight: writes happen only in the Primary region; Secondaries are read-only (single-writer). Replication is asynchronous but, over AWS's private infrastructure (storage-layer replication), typically achieves sub-1-second lag.

On region failure, promote a Secondary to standalone primary (usually within a minute). Note: to return to a global configuration post-failover, you must wait for the old Primary to recover, then reconfigure it as the new Secondary—it doesn't auto-reverse bidirectionally.

💡 Related Theory: Aurora Global's insistence on "single writer + read-only secondary" exists to eliminate distributed write conflicts by construction. If multiple regions write to the same row simultaneously, conflict resolution logic is needed, but reconciling transaction/foreign-key/constraint semantics with conflict resolution is hard (distributed consensus cost). So Aurora Global adopts the single-leader replication model—"writes in one place, reads everywhere"—keeping consistency straightforward. The opposite extreme is DynamoDB Global Tables' multi-leader. Single-leader is easy on consistency but couples write availability to the Primary; multi-leader can write anywhere but must handle conflicts—this tradeoff is the essence of choosing between these two services.

DynamoDB Global Tables—True Multi-Master and Last-Write-Wins

DynamoDB Global Tables is the opposite of Aurora Global: every region can read AND write in true multi-master fashion. Write to one region, and it auto-replicates bidirectionally to all others (usually <1 second). Write from anywhere, so write availability is high, but if two regions update the same item nearly simultaneously, conflicts arise. DynamoDB resolves this via Last Write Wins (LWW)—the write with the later timestamp wins.

🔍 Deeper: LWW is the simplest conflict-resolution strategy but carries risk. The "later" in "last write wins" is judged by wall-clock timestamp, yet in distributed systems, different nodes' clocks never sync perfectly (clock skew). If clocks drift, a write that truly happened first can be timestamped "later" and win anyway, causing silent lost updates. This limitation is why distributed systems theory uses more sophisticated resolution like vector clocks (Lamport's logical clock extension) or CRDT (Conflict-free Replicated Data Type)—merge conflicts based on causality, not clocks. DynamoDB chose LWW for simplicity/performance, so "update the same key from multiple regions simultaneously" is an anti-pattern. Partition keys by region or use it on low-conflict workloads.

📚 Case Study: The original Amazon Dynamo paper (DeCandia et al., SOSP 2007) handled shopping-cart conflicts not with LWW but by preserving both versions so applications could merge—in carts, "don't lose added items" mattered more than "latest state." This teaches: "conflict resolution needs domain knowledge." Lesson: LWW is not universal; depending on data semantics, "last wins" can mean data loss, so multi-master requires examining conflict scenarios rigorously.

S3 Cross-Region Replication—Asynchronous Object Replication

S3 CRR (Cross-Region Replication) automatically replicates objects from one bucket to another in a different region. Both buckets must enable versioning—version IDs track replication. Replication is unidirectional by default; for bidirectional, configure CRR in both directions. The replication role (IAM) needs read on source + write on destination.

• RTC (Replication Time Control): Guarantees 99.99% of objects replicate within 15 minutes (SLA). Plain CRR is best-effort.
• Selective replication: Replicate by prefix/tags, to different storage classes, to a different account.
• Pre-existing objects: CRR by default replicates only new objects created after enabling (old objects need S3 Batch Replication).

⚠️ Pitfall: CRR replicates only new objects created after versioning is enabled, a frequent gotcha. "I turned on CRR but old objects aren't in the destination bucket" is not a bug—it's design. To replicate pre-existing objects, run S3 Batch Operations (Batch Replication) separately. Also, delete marker replication is optional; deleting in one region may not delete the object in another by default.

KMS Multi-Region Key—Bridging Encryption Boundaries Across Regions

A hidden pitfall in multi-region data is encryption. When S3 CRR replicates encrypted objects to another region, or Aurora Global/DDB Global Tables replicate data, if the KMS key used for encryption doesn't exist in that region, decryption is impossible. A single-region KMS key is locked to its region and unusable elsewhere.

Multi-Region Key (MRK) solves this. Create a Primary MRK in one region, replicate Replica MRKs to others, and they share the same key material and key ID (mrk- prefix). So data encrypted with an MRK in region A can be decrypted with the replica MRK in region B—ciphertext is valid across region boundaries.

aws kms create-key --multi-region   # Create Primary MRK
aws kms replicate-key --key-id mrk-abc --replica-region us-east-1  # Replicate

💡 Related Theory: Sharing "the same key material across multiple regions" is cryptographically subtle. Normally, keys are confined to one place (minimize key exposure surface), but MRK intentionally replicates key material to gain portability of ciphertext—while key material is shared, access controls (policies, grants) are managed independently per region (same material, regional permissions). This works with envelope encryption: data is encrypted with a data key (DEK), and the DEK is encrypted with a KMS key (KEK). MRK makes the KEK layer shareable across regions, so encrypted DEKs accompanying replicated ciphertext can be decrypted in the destination region. In exams, "replicated encrypted data can't be decrypted in another region" almost always answers to "no MRK" or "missing key policy permission."

Global Accelerator vs CloudFront—Two Routes Around DNS Limitations

Two services bypass Route 53 DNS routing's limitation (slow failover due to cache TTL), but they serve different purposes.

🔍 Deeper: Global Accelerator's instant failover secret is anycast. Anycast is a routing technique advertising the same IP address from multiple edge locations worldwide (via BGP); user packets automatically route to the nearest edge. Clients only ever know two fixed IPs; if a backend region dies, GA instantly reroutes to the healthy region—regardless of client DNS cache (the IP doesn't change). Route 53, by contrast, changes the IP itself on failover, so delay is at least one DNS cache TTL. "Instant cross-region failover for non-HTTP" → GA; "global caching of static content" → CloudFront; "simple weighted/latency routing" → Route 53.

Route 53 ARC—Making DR Failover Safe by Code

Relying solely on health-check-driven failover risks misinterpreting health or partial outages, triggering wrong failover. Route 53 Application Recovery Controller (ARC) treats failover as explicit control (Routing Control)—operators or automation safely toggle "route traffic to region B now" via code/API. Concurrently, Readiness Check continuously verifies "is the Secondary region truly ready?" (capacity, configuration, replication state).

💡 Related Theory: ARC's core value is enabling safe DR drills (tested failover). A plague of DR plans: "untested failover paths fail when disasters strike" (code rot from disuse). ARC Routing Control is distributed across 5 independent region data planes, so even if the main control plane fails, failover can be toggled (control/data plane separation—fewer dependent systems on failure, per the "static stability" principle). This commercializes the Well-Architected principle: "regularly test recovery procedures." Validating this failover via FIS chaos experiments is a pattern from Day 4.

Closing Thoughts

Four pictures emerge today. First, multi-region must accept asynchronous replication due to the speed of light, and handling resulting inconsistency/conflicts is the core. Second, Route 53 is DNS-based GSLB combining 7 routing policies and quorum health checks, but DNS cache TTL is a fundamental delay; instant failover falls to anycast-based Global Accelerator. Third, data replication trades off single-leader (Aurora Global, no conflicts) against multi-leader (DDB Global Tables, LWW); LWW risks silent lost updates from clock skew. Fourth, KMS Multi-Region Key bridges encryption boundaries, letting replicated ciphertext be decrypted in other regions; S3 CRR replicates only new post-versioning objects; Route 53 ARC makes failover safe code via control/data plane separation.

Next, we explore the four DR strategies—Backup & Restore, Pilot Light, Warm Standby, Active-Active—and their RTO/RPO/cost tradeoffs.

📌 Today's Summary

Four pictures emerge from today. First, multi-region must accept asynchronous replication because of the physical limit of the speed of light, and the essence of the work is handling the resulting conflicts and inconsistency. Second, Route 53 is DNS-based GSLB combining seven routing policies with quorum health checks (consensus among many checkers prevents false positives), but DNS cache TTL is a fundamental source of delay, so instant failover falls to anycast fixed-IP Global Accelerator (CloudFront handles HTTP content caching). Third, data replication is a tradeoff between single-leader (Aurora Global — one write point, so no conflicts, secondary promote in about a minute) and multi-master (DynamoDB Global Tables, LWW conflict resolution), and LWW carries the risk of silent lost updates from clock skew. Fourth, a KMS Multi-Region Key shares the same key material across regions to give replicated ciphertext regional portability (sharing the envelope-encryption KEK), S3 CRR requires versioning and replicates only new objects, and Route 53 ARC provides a safe DR failover toggle through control/data plane separation.