Day 1 - Elastic Beanstalk and Five Deployment Policies: The True Cost of Zero Downtime

When first selecting "Deployment policy" in the Beanstalk console, five options appear equivalent: All at once, Rolling, Rolling with additional batch, Immutable, Blue-Green. The names sound similar, yet they determine whether you experience five minutes of downtime, whether your 30-minute deployment costs twice as much, and whether a single alarm instantly triggers rollback.

This article begins with why Beanstalk as a PaaS intentionally creates EC2/ALB/ASG inside customer accounts, then examines the five policies across four trade-off axes: speed, cost, downtime, and rollback. The goal isn't memorizing tables for exams but developing the operator's instinct to judge "which policy fits this release?"

Why Beanstalk Straddles PaaS and IaaS

Viewing Beanstalk merely as "Heroku for AWS" captures only half the picture. When launched January 2011, AWS already sold EC2, RDS, ELB, and SQS separately; developers complained: "Manually clicking these together in the console is exhausting." Heroku (2007) and Google App Engine (2008) dominated the PaaS market. AWS answered with Beanstalk, making an interesting choice: infrastructure remains visible within customer accounts.

Heroku hides EC2 behind "Dynos" abstraction; customers never see it. Beanstalk reversed this. aws ec2 describe-instances displays EC2; ALB, ASG, RDS all remain customer account resources. This embodies the PaaS convenience + IaaS control trade-off. In exam scenarios choosing between Beanstalk and ECS: Beanstalk fits "automate existing EC2 workloads"; ECS/Fargate fits "rewrite with container standards."

💡 Related Theory: NIST SP 800-145 defines PaaS as "customers deploy applications but don't control underlying cloud infrastructure (network, servers, OS, storage)." Beanstalk partially violates this definition—customers can SSH into EC2 and manually modify security groups. Some literature calls Beanstalk "Managed IaaS" or "Application Platform." For exams, PaaS is correct; understanding actual responsibility models as IaaS-like prevents trap answers.

🔍 Deeper Dive: Beanstalk's internals run atop CloudFormation. Running eb create internally creates a CloudFormation stack (named awseb-e-xxx-stack), which provisions EC2, ALB, ASG, SG. Running aws cloudformation describe-stack-events shows Beanstalk environment creation step-by-step. Two implications: ① Beanstalk environment problems → CloudFormation events are first-order diagnosis. ② Manually modifying Beanstalk resources in console → stack drift → overwritten on next deployment.

The Real Trade-Offs Across Five Deployment Policies

Everyone memorizes deployment policy comparison tables. The problem: when judging "which fits?" from scenarios, most focus on only two axes (downtime, cost). Reality encompasses speed, capacity, cost, rollback speed, and failure impact scope—five axes all different.

Many say "never use All at once in production," but it's actually most rational for dev/staging. No traffic impact, fastest deployment, failure = redeploy again. When operators choose policies, the key metric is "environment tier × failure cost."

Why Rolling's "Capacity Decrease" Is Dangerous

Rolling often gets pitched as "zero downtime, no extra cost—great option." In reality, overlapping with traffic spikes creates downtime-equivalent effects. With 4-instance ASG and 50% batch Rolling, only 2 instances handle traffic during deployment. If traffic spikes to 1.2x normal, response time explodes; ELB health checks start failing. Failed health checks signal ASG to "replace instances," launching replacements even of deployed instances, trapping deployment in infinite loop.

📚 Case Study: Atlassian's April 5, 2022 14-day outage—one root cause was "script error + partial capacity during deployment." Maintenance script unintentionally queued ~400 customer sites for permanent deletion; recovery required gradual deployment at 48 hours per site. Multi-stage deployment seemed safe; during disaster recovery, throughput became the bottleneck. Atlassian official post-incident review. After that, Atlassian increased auto-recovery tool parallelism 100x.

⚠️ Gotcha: Health check grace period set too short during Rolling → new instances fail health checks before warming up → deployment halts. Beanstalk default grace period: 300s. JVM apps (Spring Boot, etc.) commonly need 60-90s cold start; 600s+ is safer.

The "2x Cost" Trap of Immutable

Immutable policy: "keep old ASG, create new ASG, validate, then switch traffic." New ASG spawns same instance count as old, hence "2x cost" explanation. This 2x applies during validation only (usually 5-15m). AWS charges per hour (t/m/c series hourly; some nano/micro second-based). A 30-minute deployment bills over 1 hour of instance time.

Teams deploying 100x monthly face non-trivial costs (t3.medium 4-instance × 1 hour × 100 = ~$16/month). Operators should understand "Immutable cost scales with deployment frequency," not "Immutable is expensive."

Blue-Green URL Swap's DNS TTL Trap

Beanstalk Blue-Green creates two environments, then swaps CNAMEs via swap-environment-cnames. CloudFront or Route 53 Alias enables relatively fast switches, but clients and ISPs cache DNS, so all users reaching new environment takes TTL duration.

Beanstalk default CNAME TTL: 60s. Some ISPs violate RFC 2181, using longer TTL. Some Korean mobile carriers cache 5m; some corporate DNS resolvers cache 30m. Thus Blue-Green enables instant rollback, but actual traffic shift is gradual. Exams frequently trap: "Blue-Green = instant 100% switch." That's misleading.

💡 Related Theory: DNS TTL vs. consistency trade-off defined in RFC 1035 (1987) and RFC 2181 (1997). Short TTL enables fast propagation but burdens authoritative servers; long TTL reduces load but delays changes. AWS Global Accelerator uses BGP Anycast static IPs instead of DNS specifically to bypass this trade-off. Anycast switches via BGP routing table updates (typically seconds), not waiting for DNS TTL.

Comparison with CodeDeploy and Kubernetes Rolling Updates

Beanstalk deployment policies aren't AWS-only inventions. Comparing to other systems reveals essentials.

GCP App Engine and Azure Deployment Slot most resemble Beanstalk Blue-Green. Azure Slot Swap is more elegant—it changes routing tables at IP level, bypassing DNS TTL issues. AWS achieves similar effect via ALB Target Group weights; Beanstalk avoids this mechanism to guarantee stronger environment isolation (completely separate ASG, DB, configuration).

Kubernetes's maxSurge and maxUnavailable precisely generalize Rolling with additional batch. maxSurge=25% allows +25% temporary instances; maxUnavailable=0% forbids capacity decrease. Thus "K8s RollingUpdate with maxSurge=25%, maxUnavailable=0% = Beanstalk Rolling with additional batch 25%" behave identically.

Worker Tier: The Small Invention of SQS Daemon

Worker Tier represents Beanstalk's underrated strength. Typical Worker pattern (EC2 directly polling SQS) embeds SQS SDK, retry logic, and DLQ handling into code. Beanstalk Worker separates this as sidecar daemon.

[SQS Queue] ──┐
              │ poll
              ▼
        [SQS Daemon] ── HTTP POST localhost:80 ──→ [Web App (any language)]
              │                                          │
              │ ◄────────── HTTP 200 ─────────────────── │
              ▼
        DeleteMessage

SQS Daemon receives message, HTTP POSTs to local app (usually port 80); app returns 200 → auto-delete, 4xx/5xx → visibility timeout → retry. App code needs only one HTTP handler. This design aligns perfectly with 12-factor apps' "Process Type" (Worker and Web share code, differ in entry point).

🔍 Deeper Dive: SQS Daemon includes HTTP headers: X-Aws-Sqsd-Msgid, X-Aws-Sqsd-Queue, X-Aws-Sqsd-First-Received-At, X-Aws-Sqsd-Receive-Count. Apps can examine Receive-Count to adapt strategy by retry count (e.g., ≥3 → static analysis → immediate DLQ move). Defining periodic tasks via cron.yaml internally creates lock messages for leader election (preventing concurrent runs across instances).

RDS Externalization: Lessons Learned Only After Failure

Creating Beanstalk environment with "Add DB" option generates RDS with environment. Tempting but environment termination deletes RDS. Common prod incident: "staging cleanup accidentally deleted prod DB backup."

Recommended pattern: separate RDS into external CloudFormation/Terraform stack, inject endpoint via Beanstalk environment variables (RDS_ENDPOINT, RDS_PASSWORD). Even safer: credentials in Secrets Manager, retrieve via IAM permission.

# Recommended: inject external RDS endpoint via environment variable
aws elasticbeanstalk update-environment \
  --environment-name MyApp-prod \
  --option-settings \
    Namespace=aws:elasticbeanstalk:application:environment,OptionName=DB_HOST,Value=mydb.xxx.rds.amazonaws.com \
    Namespace=aws:elasticbeanstalk:application:environment,OptionName=DB_SECRET_ARN,Value=arn:aws:secretsmanager:...:secret:db-creds

📚 Case Study: 2017 Travis CI experienced production environment migration data loss; partial cause was "unclear DB separation by environment → staging cleanup affected prod." Subsequently, many adopted: always separate DB from environment stack. AWS Well-Architected Framework Reliability Pillar explicitly codifies: "data lifecycle decoupled from compute lifecycle."

.ebextensions and .platform/hooks

Beanstalk OS-level automation evolved through two mechanisms.

.ebextensions/*.config predates Amazon Linux 2, defining packages, environment variables, container_commands in YAML/JSON. Amazon Linux 2 (Platform v3) changed some behaviors.

.platform/hooks/ introduced with Amazon Linux 2, executing shell scripts by deployment stage based on directory structure.

.platform/
  hooks/
    prebuild/     # before app build
    predeploy/    # before deployment
    postdeploy/   # after deployment
  confighooks/
    prebuild/
    predeploy/
    postdeploy/

Most common migration trap: container_commands. Platform v2 executed container_commands from staging directory; v3 requires migration to prebuild/predeploy. Not simple lift-and-shift.

Summary

Two key takeaways: First, Beanstalk compromises PaaS convenience with IaaS control, exposing EC2, ALB, RDS in customer accounts. Second, five deployment policies aren't simple "options" but coordinates across a five-axis trade-off: speed, capacity, cost, rollback, and failure impact.

Next, we'll examine CodeDeploy—unlike Beanstalk bundling environments, it focuses on "code-only deployment." AppSpec hook execution order, Lambda Canary weight mechanics, ECS Blue-Green Target Group switching—see how different compute platforms implement the same deployment abstraction.