Day 1 - CloudFormation: Infrastructure as Code from the Operator's Perspective

In 2011, Netflix released "Chaos Monkey"—a tool that randomly terminates production instances. This tool could exist because Netflix managed infrastructure as code. When instances died, CloudFormation (or the IaC tool of that time) automatically recreated identical instances. Chaos Monkey would have been a disaster with manually configured infrastructure.

CloudFormation is AWS's official IaC (Infrastructure as Code) service. Launched February 2011, it started with JSON templates and now defaults to YAML. In SOA-C02, CloudFormation isn't simply an "infrastructure creation tool" but rather an operational platform where "operators safely manage changes, detect state drift, and auto-recover on failure."

History of IaC and CloudFormation's Position

IaC concepts systematized in 2006 by Mark Burgess (CFEngine author) in "In Search of Certainty." The core principle: "describe the system's desired state declaratively." Chef (2009), Puppet Enterprise (2011), Ansible (2012), and Terraform (2014) followed. Terraform, announced 2014 by HashiCorp's Mitchell Hashimoto with "Write, Plan, Apply" workflow, became industry standard.

CloudFormation's unique position is being completely AWS-native. While other IaC tools treat AWS as one provider, CloudFormation is AWS itself. When AWS launches new services, CloudFormation resource types appear within days or weeks. Terraform requires community or HashiCorp Provider updates for new services.

IaC Tool Advanced Comparison:

💡 Related Theory: CloudFormation's "Declarative" approach resembles set theory in mathematics. "This set (infrastructure) should contain these elements (resources)"—the system calculates difference between current and goal state and converges. Contrasts with Imperative Shell scripts describing "Step 1: Create EC2, Step 2: Create SG...". Declarative's core is Idempotency: applying same template multiple times yields identical results. This concept traces to Dijkstra's "Separable Abstraction" principle, even to 1970s hardware machine independence debates—the software version.

🔍 Deeper Dive: CloudFormation engine internally uses DAG (Directed Acyclic Graph) for resource dependency analysis. When dependencies defined via !Ref and !GetAtt, CFn performs topological sorting, determining which resources to create first and which can parallel. Explicit DependsOn captures implicit dependencies (e.g., IAM permission propagation delay). This DAG approach mirrors GNU Make Makefile dependency analysis, Apache Airflow DAG scheduling—all founded in same CS theory.

CloudFormation Engine Internal Operation

When CloudFormation receives create-stack:

Step 1: Template Parsing and Validation

• JSON/YAML parsing → internal representation
• 10-section structure validation
• Intrinsic Functions interpretation (recursive evaluation)
• Circular reference detection (blocks A → B → A)

Step 2: Dependency Graph Construction

• Extract implicit dependencies from !Ref and !GetAtt
• Add explicit dependencies from DependsOn
• Validate DAG (no cycles)
• Topological sort determines creation order

Step 3: Parallel Resource Creation

• Create resources without dependencies simultaneously
• Wait for dependent resources' predecessors
• Poll each resource API call for completion
• Emit CloudWatch Events stream

Step 4: CreationPolicy Signal Waiting (if specified)

• Wait to receive Count cfn-signal signals
• Timeout → failure handling
• Success signal → proceed to next step

Step 5: Finalize Stack State

• All resources successful → CREATE_COMPLETE
• Any resource fails → start ROLLBACK_IN_PROGRESS

CloudFormation Internal Processing Flow:

Template YAML
    │
    ▼
┌─────────────────────────────────────────┐
│  CFn Parser                             │
│  - JSON/YAML → IR (Internal Repr.)      │
│  - Intrinsic Function evaluation        │
│  - Pseudo Parameter substitution        │
└──────────────────┬──────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────┐
│  Dependency Analyzer (DAG Builder)      │
│  !Ref / !GetAtt → implicit dependencies │
│  DependsOn → explicit dependencies      │
│  Topological Sort → execution plan      │
└──────────────────┬──────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────┐
│  Resource Executor (parallel)           │
│  ┌─────────┐  ┌─────────┐             │
│  │ IAM Role│  │  VPC    │  (parallel)  │
│  └────┬────┘  └────┬────┘             │
│       └─────┬──────┘                   │
│             ▼                          │
│        ┌─────────┐                     │
│        │EC2 inst │  (after both above)  │
│        └─────────┘                     │
└─────────────────────────────────────────┘

Template 10 Sections: Only What Operators Need to Know

CloudFormation Template has up to 10 sections. Only Resources is required; rest optional. This fact repeats on exams.

AWSTemplateFormatVersion: '2010-09-09'  # Optional (current only value)
Description: 'My application stack'     # Optional
 
Metadata:           # Optional - console UI hints, group parameters
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - Label:
          default: "Database Configuration"
        Parameters: [DBInstanceClass, DBPassword]
 
Parameters:         # Optional - input values at stack creation
  EnvType:
    Type: String
    AllowedValues: [dev, staging, prod]
    Default: dev
    Description: "Deployment environment"
 
Mappings:           # Optional - lookup table (Region → AMI etc)
  RegionAMIMap:
    ap-northeast-2:
      AMI: ami-0c9c942bd7bf113a2
    us-east-1:
      AMI: ami-0c55b159cbfafe1f0
 
Conditions:         # Optional - conditional resources/properties
  IsProd: !Equals [!Ref EnvType, prod]
  IsNotDev: !Not [!Equals [!Ref EnvType, dev]]
 
Transform:          # Optional - SAM, macro transformation
  - AWS::Serverless-2016-10-31
 
Resources:          # Required - actual AWS resource definitions
  MyBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub 'myapp-${EnvType}-${AWS::AccountId}'
 
Outputs:            # Optional - expose to other stacks or view in console
  BucketArn:
    Value: !GetAtt MyBucket.Arn
    Export:
      Name: !Sub '${AWS::StackName}-BucketArn'
 
Rules:              # Optional - cross-validation between Parameters
  MustUseProdDB:
    RuleCondition: !Equals [!Ref EnvType, prod]
    Assertions:
      - Assert: !Equals [!Ref DBInstanceClass, db.r5.large]
        AssertDescription: "Production must use db.r5.large"

Parameters: Sophisticated Input Control

Parameters make Templates reusable. Same template creates dev, staging, prod environments.

Complete Parameter Type List:

Dynamic References (fetch external values without Parameter):

# Direct reference from Secrets Manager
DBPassword:
  !Sub '{{resolve:secretsmanager:prod/db/password:SecretString:password}}'
 
# Direct reference from Parameter Store
DBHost:
  !Sub '{{resolve:ssm:/myapp/prod/database/host}}'
 
# Reference encrypted value from Parameter Store
DBPassword2:
  !Sub '{{resolve:ssm-secure:/myapp/prod/database/password}}'

🔍 Deeper Dive: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id> type automatically fetches latest AMI ID from AWS-maintained SSM parameter path (/aws/service/ami-amazon-linux-latest/...). These paths auto-update when AWS publishes new AMIs. Same template deployed years later always uses latest Amazon Linux 2. Avoids "deprecated AMI ID" problems from hardcoding/Mappings. Caveat: without stack redeployment, SSM parameter changes don't reflect—and Drift Detection doesn't catch this gap.

Intrinsic Functions: Operator Essential

Nested Function Examples (practical patterns):

# Auto-select first AZ subnet
SubnetId: !Select
  - 0
  - !GetAZs ''
 
# Conditional instance type + Sub nesting
InstanceType: !If
  - IsProd
  - !Sub 'r5.${ProdInstanceSize}'
  - t3.micro
 
# Auto-split CIDR (auto-create VPC subnets)
CidrBlock: !Select
  - 0
  - !Cidr [!GetAtt VPC.CidrBlock, 6, 12]
  # Split VPC CIDR into 6 /20 subnets, select first

Stack Lifecycle: Know All States for Troubleshooting

Stack State Transition Diagram:

CREATE_IN_PROGRESS ──────────────► CREATE_COMPLETE
        │
        └──(failure)──► ROLLBACK_IN_PROGRESS ──► ROLLBACK_COMPLETE
                                                    │
                                                    └── [Update impossible, delete only]

CREATE_COMPLETE ──(update-stack)──► UPDATE_IN_PROGRESS ──► UPDATE_COMPLETE
                                            │
                                            └──(failure)──► UPDATE_ROLLBACK_IN_PROGRESS
                                                                │
                                                                ├──(success)──► UPDATE_ROLLBACK_COMPLETE
                                                                │
                                                                └──(failure)──► UPDATE_ROLLBACK_FAILED
                                                                                    │
                                                                                    └── continue-update-rollback needed

UPDATE_COMPLETE ──(delete-stack)──► DELETE_IN_PROGRESS ──► DELETE_COMPLETE
                                            │
                                            └──(failure)──► DELETE_FAILED
                                                             │
                                                             └── Use retain option, exclude problem resource, retry delete

Important State Descriptions:

# Delete ROLLBACK_COMPLETE stack
aws cloudformation delete-stack --stack-name failed-stack
aws cloudformation wait stack-delete-complete --stack-name failed-stack
 
# Recover UPDATE_ROLLBACK_FAILED
aws cloudformation continue-update-rollback \
  --stack-name my-stack \
  --resources-to-skip LogicalResourceId1 LogicalResourceId2
 
# Recover DELETE_FAILED: preserve problem resource
aws cloudformation delete-stack \
  --stack-name my-stack \
  --retain-resources ProblematicS3Bucket ProblematicENI
 
# View stack events to find failure cause
aws cloudformation describe-stack-events \
  --stack-name my-stack \
  --query 'StackEvents[?ResourceStatus==`CREATE_FAILED` || ResourceStatus==`UPDATE_FAILED`].[Timestamp,LogicalResourceId,ResourceStatusReason]' \
  --output table

⚠️ Pitfall: ROLLBACK_COMPLETE state is initial stack creation's failure final state. Can't update-stack here; must delete-stack then recreate. Attempting update-stack errors with "Stack is not in a valid state for UpdateStack." continue-update-rollback is exclusive to UPDATE_ROLLBACK_FAILED, not ROLLBACK_COMPLETE. Exam question "recover stack from ROLLBACK_COMPLETE?" always answers "delete and recreate."

Resource Properties: DeletionPolicy, UpdateReplacePolicy, CreationPolicy

DeletionPolicy: How to handle resources when Stack is deleted

MyDatabase:
  Type: AWS::RDS::DBInstance
  DeletionPolicy: Snapshot    # Create snapshot before deleting instance
  Properties: ...
 
MyBucket:
  Type: AWS::S3::Bucket
  DeletionPolicy: Retain       # Keep bucket when stack deleted
  Properties: ...
 
MyCacheCluster:
  Type: AWS::ElastiCache::ReplicationGroup
  DeletionPolicy: Delete       # Default: delete resource with stack
  Properties: ...

UpdateReplacePolicy: When Stack update triggers resource replacement

UpdateReplacePolicy determines old resource handling when Stack update causes Replacement. DeletionPolicy acts on Stack deletion; UpdateReplacePolicy on update replacement.

MyDatabase:
  Type: AWS::RDS::DBInstance
  DeletionPolicy: Retain
  UpdateReplacePolicy: Snapshot  # Snapshot old DB when update replaces it
  Properties:
    DBInstanceClass: db.t3.medium  # → Change to db.r5.large triggers RDS replacement
    MultiAZ: false                  # → Change to true triggers RDS replacement

Resource Replacement (Replacement) Trigger Examples:

📚 Case Study: 2022, e-commerce company F had CloudFormation update accident. RDS instance unexpectedly replaced. Changed DB class from db.t3.medium to db.r5.xlarge while changing MultiAZ: false to true. AWS treated MultiAZ change as Replacement; UpdateReplacePolicy defaulted to Delete, so old DB deleted without snapshot. Lost 4 hours data from recent manual snapshot. Later implemented standard: all RDS set DeletionPolicy: Snapshot and UpdateReplacePolicy: Snapshot; Change Set review process mandates check Replacement: True resources.

CreationPolicy: Wait for resource creation completion signal

Pattern where CloudFormation waits until EC2 User Data script completes and application is ready.

WebServer:
  Type: AWS::EC2::Instance
  CreationPolicy:
    ResourceSignal:
      Count: 1
      Timeout: PT15M  # ISO 8601 Duration: 15-min wait
  Properties:
    ImageId: !Ref AmiId
    UserData:
      Fn::Base64: !Sub |
        #!/bin/bash
        yum update -y
        yum install -y httpd
        systemctl start httpd
        systemctl enable httpd
        echo "<h1>Hello from ${AWS::StackName}</h1>" > /var/www/html/index.html
        # Signal CloudFormation on completion
        /opt/aws/bin/cfn-signal \
          --exit-code $? \
          --stack ${AWS::StackName} \
          --resource WebServer \
          --region ${AWS::Region}

When cfn-signal sends exit-code 0, CloudFormation proceeds to CREATE_COMPLETE. exit-code 1 or no signal within Timeout = creation failure.

cfn-signal Debugging Patterns:

# Check EC2 system log on cfn-signal failure (console)
aws ec2 get-console-output --instance-id i-xxxx
 
# View UserData execution log (connect via SSM Session Manager)
cat /var/log/cfn-init.log
cat /var/log/cfn-init-cmd.log
cat /var/log/cloud-init-output.log
 
# Check cfn-signal success in CloudFormation events
aws cloudformation describe-stack-events \
  --stack-name my-stack \
  --query 'StackEvents[?ResourceStatus==`CREATE_FAILED`].[LogicalResourceId,ResourceStatusReason]'

⚠️ Pitfall: cfn-signal Timeout uses ISO 8601 Duration format. PT15M (15 min), PT1H (1 hour), PT1H30M (1.5 hours). Most mistakes: timeout too short. OS package update (yum update -y) alone takes 5-10 min depending on region/network. Recommend minimum PT15M; heavy software (Java, Node.js build) needs PT30M+. Timeout event shows "WaitCondition timed out" or "Failed to receive N resource signal(s) within the specified duration."

Auto Scaling Group's UpdatePolicy (Rolling Update):

WebASG:
  Type: AWS::AutoScaling::AutoScalingGroup
  CreationPolicy:
    ResourceSignal:
      Count: !Ref DesiredCapacity
      Timeout: PT20M
  UpdatePolicy:
    AutoScalingRollingUpdate:
      MaxBatchSize: 2              # Replace 2 at a time
      MinInstancesInService: 1     # Always keep 1 in service
      PauseTime: PT5M              # Wait 5 min between batches (WaitOnResourceSignals=false)
      WaitOnResourceSignals: true  # true→wait for cfn-signal, false→use PauseTime
      SuspendProcesses:            # Suspend ASG processes during rolling update
        - HealthCheck
        - ReplaceUnhealthy
        - AZRebalance
  Properties:
    MinSize: 2
    MaxSize: 6
    DesiredCapacity: !Ref DesiredCapacity

AutoScalingRollingUpdate Key Parameter Interactions:

💡 Related Theory: CreationPolicy and cfn-signal pattern mirrors distributed systems' "readiness probe" concept. Like Kubernetes readinessProbe, don't forward traffic before app signals readiness. In CloudFormation, this signal is cfn-signal; Stack stays CREATE_IN_PROGRESS, won't proceed until signaled. Rolling Update with WaitOnResourceSignals=true repeats this per batch, confirming "is this instance really ready?" Michael Fowler's "Blue-Green Deployment" replaces entire cluster; AutoScalingRollingUpdate's gradual replacement resembles "Canary Release."

Stack Policy: Block Critical Resource Updates

Stack Policy is firewall preventing accidental modification/deletion of critical resources during Stack update. Unlike IAM policy controlling human API access, Stack Policy controls CloudFormation operations on resources.

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "Update:*",
      "Principal": "*",
      "Resource": "*"
    },
    {
      "Effect": "Deny",
      "Action": ["Update:Replace", "Update:Delete"],
      "Principal": "*",
      "Resource": "LogicalResourceId/ProdDatabase"
    },
    {
      "Effect": "Deny",
      "Action": "Update:*",
      "Principal": "*",
      "Resource": "LogicalResourceId/ProdCacheCluster"
    }
  ]
}

Stack Policy Action Types:

# Set Stack Policy
aws cloudformation set-stack-policy \
  --stack-name my-prod-stack \
  --stack-policy-body file://stack-policy.json
 
# Override Stack Policy temporarily (admin permission required)
aws cloudformation update-stack \
  --stack-name my-prod-stack \
  --template-body file://template.yaml \
  --stack-policy-during-update-body '{"Statement":[{"Effect":"Allow","Action":"Update:*","Principal":"*","Resource":"*"}]}'
 
# View current Stack Policy
aws cloudformation get-stack-policy --stack-name my-prod-stack

💡 Related Theory: Stack Policy evaluation follows IAM's explicit Deny precedence rule. Default differs: IAM defaults deny all; Stack Policy defaults allow all. When no Stack Policy, all updates allowed. Once set, only explicit Allow statements pass; unlisted resources denied. To deny specific resource, must add "allow all resources" statement. This pattern mirrors firewall rules' "Allow All, Deny Specific."

Pseudo Parameters: Always-Available Built-In Variables

# Available Pseudo Parameters
${AWS::AccountId}     # Current AWS account ID
${AWS::Region}        # Current region (e.g., ap-northeast-2)
${AWS::StackId}       # Stack ARN
${AWS::StackName}     # Stack name
${AWS::URLSuffix}     # Domain suffix (amazonaws.com or CN: amazonaws.com.cn)
${AWS::Partition}     # aws, aws-cn, aws-us-gov
${AWS::NoValue}       # Remove property conditionally
 
# Usage examples
BucketName: !Sub 'myapp-${AWS::AccountId}-${AWS::Region}-${EnvType}'
# → myapp-123456789012-ap-northeast-2-prod
 
# AWS::NoValue usage: conditionally remove property
MultiAZ: !If
  - IsProd
  - true
  - !Ref AWS::NoValue  # Dev: remove MultiAZ (use default false)

Partition Usage (GovCloud, China regions):

# Build correct ARN regardless of region
PolicyArn: !Sub 'arn:${AWS::Partition}:iam::${AWS::AccountId}:policy/MyPolicy'
# us-east-1: arn:aws:iam::123456789012:policy/MyPolicy
# cn-north-1: arn:aws-cn:iam::123456789012:policy/MyPolicy
# us-gov-east-1: arn:aws-us-gov:iam::123456789012:policy/MyPolicy

Troubleshooting Patterns: Operator's Daily Work

Disable Rollback for Debugging:

# On creation failure, preserve failed resources for inspection
aws cloudformation create-stack \
  --stack-name debug-stack \
  --template-body file://template.yaml \
  --on-failure DO_NOTHING \
  --parameters ...
 
# Disable rollback on update
aws cloudformation update-stack \
  --stack-name my-stack \
  --template-body file://template.yaml \
  --disable-rollback

Caution: DO_NOTHING/--disable-rollback continues charging; clean up after debugging.

Template Validation:

# Validate template syntax (AWS server-side)
aws cloudformation validate-template \
  --template-body file://template.yaml
 
# Local validation with cfn-lint (more detailed)
pip install cfn-lint
cfn-lint template.yaml
 
# Rule-specific validation (SOA exam-related)
cfn-lint template.yaml --include-checks W  # Include Warnings
cfn-lint template.yaml -r E3002            # Specific rule only

CAPABILITY Flags:

# Required for Templates with IAM resources
aws cloudformation create-stack \
  --capabilities CAPABILITY_IAM         # Unnamed IAM resources
  --capabilities CAPABILITY_NAMED_IAM   # Named IAM resources
  --capabilities CAPABILITY_AUTO_EXPAND # Transform (SAM/Macro)
 
# Multiple flags possible
aws cloudformation create-stack \
  --capabilities CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND

CAPABILITY Absence Error:

InsufficientCapabilities: Requires capabilities : [CAPABILITY_NAMED_IAM]

Occurs when Template contains IAM resources but CAPABILITY flag missing.

📚 Case Study: 2023, fintech startup G's CI/CD suddenly failed CloudFormation deploy. Error: "InsufficientCapabilities". Added AWS::IAM::Role but pipeline lacked --capabilities CAPABILITY_IAM. Fixed easily but production deploy delayed 30 min. Now pipeline scripts default to --capabilities CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND. Permitting extra capabilities risks nothing; CFn only affects IAM resource creation.

Termination Protection and Operational Governance

# Prevent accidental stack deletion
aws cloudformation update-termination-protection \
  --stack-name my-prod-stack \
  --enable-termination-protection
 
# Check Termination Protection status
aws cloudformation describe-stacks \
  --stack-name my-prod-stack \
  --query 'Stacks[0].EnableTerminationProtection'
 
# Must disable before deletion
aws cloudformation update-termination-protection \
  --stack-name my-prod-stack \
  --no-enable-termination-protection

Attempting to delete with Termination Protection enabled:

TerminationProtection: Stack [my-prod-stack] cannot be deleted while TerminationProtection is enabled

💡 Related Theory: Termination Protection is a Human Error defense layer. Netflix's "Defense in Depth" culture ensures single mistakes don't cause irreversible damage—multiple confirmation stages. CloudFormation operational governance's 3 layers: (1) Stack Policy—CFn engine blocks specific resource updates, (2) Termination Protection—blocks stack deletion, (3) IAM Permission Boundary—user/role can't call CFn delete API. Each defends different attack surface.

🔍 Deeper Dive: Most dangerous CloudFormation situation: performing Replacement unknowingly. EC2 ImageId change, RDS MultiAZ change, DynamoDB TableName change all trigger Replacement. Change Set displays each resource's change type: Replacement: True / False / Conditional. Conditional means runtime-determined; e.g., EC2 InstanceType change decides Replacement by EBS optimization. Operators must review Change Set's Replacement field; if RDS/DynamoDB shows True, verify UpdateReplacePolicy: Snapshot.

Complete Workflow Picture

CloudFormation Operator Workflow
============================================================

[Developer/Operator]
    │ Write Template (YAML)
    ▼
[cfn-lint / validate-template]  ← Local+remote validation
    │
    ▼
[create-change-set]  ← Check for Replacement=True resources
    │
    ▼
[execute-change-set]
    │
    ▼
┌─────────────────────────────────────────┐
│         CloudFormation Engine           │
│                                         │
│  1. Parse and validate template         │
│  2. Analyze DAG dependencies            │
│  3. Parallel resource create/update/delete
│  4. Check Stack Policy                  │
│  5. Wait for CreationPolicy signals     │
│  6. Generate event stream               │
└─────────────────────┬───────────────────┘
                      │
          ┌───────────┴───────────┐
          │ Success               │ Failure
          ▼                       ▼
    UPDATE_COMPLETE         UPDATE_ROLLBACK_IN_PROGRESS
                                  │
                          UPDATE_ROLLBACK_COMPLETE
                          (restore to previous state)

Event tracking:
aws cloudformation describe-stack-events --stack-name xxx
  → Real-time resource state changes