Day 3 - The Anatomy of a VPC: What Route Tables Really Decide

Secondary CIDRs and the Egress-Only IGW

A VPC can additionally attach up to 4 secondary CIDRs (added in 2017). However, a secondary is only possible within the same RFC 1918 range as the primary, and is typically used to expand when IPs run short.

IPv6 is a separate track. AWS allocates a free /56 IPv6 block to every VPC, but IPv6 traffic does not go through a NAT Gateway — it goes through an Egress-Only Internet Gateway (NAT was originally a mechanism to solve IPv4 address scarcity, so it's unnecessary for IPv6).

The True Identity of a Subnet: AZ Affinity and 1:1 Route Table Mapping

A subnet is bound to a single AZ and has one route table attached. Multiple subnets can share the same route table, but one subnet cannot have two route tables.

VPC (10.0.0.0/16)
├── ap-northeast-2a
│   ├── Public Subnet  10.0.1.0/24  → Route Table: PublicRT
│   └── Private Subnet 10.0.11.0/24 → Route Table: PrivateRT-A
├── ap-northeast-2b
│   ├── Public Subnet  10.0.2.0/24  → Route Table: PublicRT
│   └── Private Subnet 10.0.12.0/24 → Route Table: PrivateRT-B
└── ap-northeast-2c
    ├── Public Subnet  10.0.3.0/24  → Route Table: PublicRT
    └── Private Subnet 10.0.13.0/24 → Route Table: PrivateRT-C

The distinction between a Public Subnet and a Private Subnet is not a separate toggle in the console — it is simply whether the route table points to an IGW or a NAT GW.

💡 Related theory: Route tables operate on the longest prefix match algorithm. If two routes exist for the same packet destination, the more specific prefix (longer mask) wins. Example: with 10.0.0.0/16 → VPC local + 10.0.5.0/24 → TGW, a packet for 10.0.5.10 goes to the TGW. This is the fundamental principle of internet routing protocols like BGP and IS-IS, and AWS route tables follow the same principle.

Per-AZ Separation: NAT Gateways Are AZ-local

A commonly missed trap. A NAT Gateway is an AZ-local resource, so the proper approach is to place one per AZ. If you route every Private Subnet to a single AZ's NAT Gateway, all outbound traffic is cut off when that AZ goes down.

✗ Wrong design:
   Private Subnet A → NAT GW (AZ-a)
   Private Subnet B → NAT GW (AZ-a)  ← If AZ-a dies, B's outbound is also cut
   Private Subnet C → NAT GW (AZ-a)

✓ Correct design:
   Private Subnet A → NAT GW (AZ-a)
   Private Subnet B → NAT GW (AZ-b)
   Private Subnet C → NAT GW (AZ-c)

📚 Case study: In 2017, a fintech startup ran production with only a single-AZ NAT Gateway. One night, a power outage in AZ-a killed the NAT Gateway, cutting off all external API calls (Stripe, Twilio, etc.) from every Private Subnet. The EC2 instances in the same AZ couldn't fail over automatically either, resulting in 30 minutes of downtime. They then switched to per-AZ NAT Gateways + Multi-AZ ASG. Operating costs went up by $0.045/hour × 3 AZs = an additional $97.2/month — negligible compared to the cost of downtime.

Security Group vs NACL: The Two Most Frequently Confused

The most frequently confused point is stateful vs stateless. SG being stateful means the return of a packet that came inbound is automatically allowed without checking outbound rules. Conversely, NACLs are stateless, so return traffic must also be explicitly allowed in the outbound rules.

[Client] → TCP SYN  → [EC2:80]
[Client] ← TCP SYN-ACK ← [EC2:80]   ← SG: auto-allowed, NACL: outbound 1024-65535 must be explicit
[Client] → TCP ACK  → [EC2:80]

🔍 Deeper dive: The NACL return port range is the ephemeral port range. Linux uses 32768–60999 and Windows uses 49152–65535, but AWS recommends the safe full range of 1024-65535. If the NACL outbound only allows 80 and 443, the return SYN-ACK arrives on an ephemeral port and gets blocked — this is a common cause of the "the SG is fine but packets aren't going through" phenomenon.

💡 Related theory: The stateful firewall is a concept first commercialized by Check Point in FireWall-1 in 1992. Traditional stateless packet filters had to inspect every packet individually, but stateful firewalls maintain a connection table keyed by the TCP 5-tuple (src IP, src port, dst IP, dst port, protocol) to automatically match return packets. AWS SGs work on the same principle. Note that the connection table consumes memory, so there is a limit to how many connections a single EC2 instance can maintain concurrently (typically hundreds of thousands).

🎯 Scenario: "An EC2 instance built from a custom AMI can't be pinged from outside. The SG has ICMP All Allow. What's the problem?" — The answer is usually (1) the NACL blocks ICMP, (2) the OS firewall (iptables/firewalld) blocks it, or (3) 169.254.169.254 going to instance metadata rejects ICMP (a security feature). A missing NACL rule is the most common.

VPC Endpoints: The Two Branches of Private Traffic

By default, going from EC2 to S3 traverses the internet (IGW). Even for S3 in the same region, going through the IGW means (1) cost, (2) security risk, (3) increased latency. So you create a private path with a VPC Endpoint.

🔍 Deeper dive: A Gateway Endpoint adds a special routing entry called a prefix list to the route table. It is expressed like pl-12345 and internally auto-tracks and updates all IP ranges for S3 and DynamoDB. An Interface Endpoint, by contrast, uses the PrivateLink mechanism to create an ENI directly in the VPC, and AWS automatically hijacks DNS with a Route 53 Private Hosted Zone (answering s3.ap-northeast-2.amazonaws.com with the ENI IP).

📚 Case study: A financial firm discovered that all EC2 → S3 traffic was leaving via the IGW. Monthly data transfer costs exceeded $30,000. After adding a Gateway Endpoint, the same traffic became free. It also resolved the issue where the security team had been flagged in PCI-DSS audits because "S3 traffic traversed the internet." Trade-off: Interface Endpoints (100+ services) have an hourly cost, so with low traffic, IGW + NAT can be cheaper. The break-even point is typically 100–200 GB per month.

PrivateLink and VPC Endpoint Services

PrivateLink is a feature to "expose a service I built to other VPCs." If you put a service behind an NLB and register it as an Endpoint Service, other accounts and VPCs can access it via an Interface Endpoint. It is the standard way for SaaS companies to make their service usable inside customer VPCs.

[Provider VPC]              [Consumer VPC]
  NLB → Service              Interface Endpoint
   ↑                              ↓
   └── PrivateLink ──────────────┘

Snowflake, Datadog, and MongoDB Atlas all connect to customer VPCs via PrivateLink.

DNS and Hostnames: Name Resolution Inside the VPC

A VPC has two DNS options.

• enableDnsSupport (default true): allows use of the AWS DNS server (x.x.x.2) inside the VPC
• enableDnsHostnames (default true for default VPC): gives EC2 instances a public DNS name like ec2-1-2-3-4.ap-northeast-2.compute.amazonaws.com

⚠️ Trap: When you create a custom VPC, enableDnsHostnames defaults to false. If you don't turn it on, Private Hosted Zones (Route 53) won't work, or the automatic DNS hijack of Interface Endpoints won't work. In Pro scenarios, half of the "I created the Endpoint but EC2 still gets responses via public DNS" problems come down to this toggle.

Flow Logs: Making Invisible Packets Visible

VPC Flow Logs record the 5-tuple and outcome (ACCEPT/REJECT) of every packet traversing an ENI. They can be delivered to CloudWatch Logs or S3.

2 123456789012 eni-abc 10.0.1.5 192.168.1.10 8080 443 6 20 4500 1612345678 1612345738 ACCEPT OK
^             ^         ^         ^            ^    ^   ^ ^  ^    ^          ^          ^      ^
version       account   src       dst          spt  dpt p pkt byte start      end       action status

🔍 Deeper dive: Flow Logs record all traffic, not a sample. However, they do not capture packet contents (payload) — only metadata. They also aggregate at 1-minute intervals, so it's not real-time monitoring. They're suited for post-incident security analysis and traffic pattern analysis. A prime use case is GuardDuty analyzing Flow Logs to detect anomalous traffic.

🎯 Scenario: "A company suspects abnormal outbound traffic in its production VPC. How to diagnose?" — Answer: enable Flow Logs → analyze with Athena/CloudWatch Insights → GuardDuty does automatic pattern matching. Capturing directly with tcpdump on EC2 would mean inspecting 100 instances one by one — a huge operational burden.

SG References and Cross-VPC Communication

In SG inbound/outbound rules, you can specify the source/destination as another SG ID instead of a CIDR.

SG-Web (attached to ALB): inbound 80/443 from 0.0.0.0/0
SG-App (attached to EC2): inbound 8080 from sg-web   ← SG ID reference instead of CIDR
SG-DB  (attached to RDS): inbound 3306 from sg-app   ← chained SG reference

This pattern is the standard for 3-tier architectures. Even as EC2 instances scale up, every ENI belonging to SG-Web is automatically allowed access on 8080, so no IP management is needed.

📚 Case study: A company operated an ALB → EC2 → RDS structure with CIDR-based SGs and had to update the SGs every time EC2 IPs changed due to Auto Scaling. Switching to SG ID references eliminated the operational burden. Note that SG ID references only work within the same region and same VPC or Peered VPCs, and do not work across a TGW (some scenarios became possible in 2022, but with constraints).

Wrapping Up

A VPC is not a "box inside a box" — it is the cooperation of 8 decision points. For a single packet to flow, the SG, NACL, Route Table, IGW/NAT/Endpoint, and DNS must all agree. Manage CIDR company-wide with IPAM from the first stroke, one NAT GW per AZ, Gateway Endpoints for S3 and DynamoDB, Interface Endpoints for the other 100+ services, and SG ID references for SGs — these five principles cover 80% of the VPC scenarios on the Pro exam.

In the next article we look at EC2, EBS, ELB, and Auto Scaling — the things that actually do computing on top of this network — at Pro depth. In particular, the four Auto Scaling policy types come up on nearly every exam, so etch them into your head in advance.