Day 2 - CodeDeploy: Making the Riskiest Moment, Deployment, Safe

When is the riskiest moment for a service in operation? A traffic surge? A database failure? Google's SRE book explains that about 70% of service outages come from changes to a running system. So the riskiest moment is "right after a deployment." The reason is simple: the very fact that a deployment has finished means "changes that were invisible until now have all been exposed to users at once."

CodeDeploy is the tool that makes that risky moment safe. It carries out a deployment so that "you don't change everything at once, but little by little, verifying as you go, and able to revert immediately if a problem occurs." CodeDeploy is a major area on the DVA-C02 exam. In particular you need to know the differences by deployment target (EC2 vs Lambda vs ECS), the distinction between Blue/Green, Canary, and Linear, and the order of the lifecycle hooks in appspec.yml.

CodeDeploy's Design Philosophy: Treating Deployment as "Orchestration"

The problem CodeDeploy tries to solve is not "copy files to a server" but "safely coordinate the state transitions that happen across multiple nodes." This difference matters. Simple file copying ends with rsync or scp, but a production deployment is an ordered job of seven or more steps: ① briefly take traffic out, ② stop the existing process, ③ fetch the new files, ④ install dependencies, ⑤ start the process, ⑥ run a health check, ⑦ put traffic back in. If even one step fails, you must not move on to the next, and the steps must proceed in a consistent order across multiple instances.

CodeDeploy models each deployment as a state machine. Each instance has the states Pending → InProgress → Succeeded/Failed, and lifecycle hooks become the trigger of each state transition. What sets CodeDeploy apart is that it treats "running user-defined scripts at each hook" as a first-class citizen.

The CodeDeploy Agent polls the CodeDeploy control plane from the EC2 instance every minute ("Is there a new deployment?"). This is a short-polling model rather than HTTP long-polling, for the reasons of ① simplicity of the agent implementation, ② spreading the load on AWS servers, and ③ resilience to temporary network disconnections. The downside is "an average 30-second delay from the deployment start command to the actual start." The answer to the question "Why doesn't CodeDeploy start immediately?" is this polling model.

The Fundamental Difference Between the Three Deployment Targets

CodeDeploy supports three targets, EC2/on-premises, Lambda, and ECS, and their internal behavior is completely different. They use the same name "CodeDeploy," but they are actually closer to three separate services.

⚠️ Trap: A question like "How do you use an In-Place deployment on ECS?" is a trap. With CodeDeploy, ECS supports only Blue/Green. ECS's own rolling update feature does not use CodeDeploy; it is a separate mechanism that the ECS service performs directly. If "ECS Blue/Green with CodeDeploy" appears on the exam it is a correct answer, but "ECS In-Place with CodeDeploy" does not exist.

The EC2 Deployment Lifecycle: Exactly How Do the 10 Steps Flow?

The core of an EC2 deployment is the lifecycle hooks. The CodeDeploy Agent runs each step in order, and if a user-defined script fails in any step, the entire deployment is marked as failed.

In-Place deployment (7 steps)
====================
1. ApplicationStop        [user script]   Stop the existing app
2. DownloadBundle         [agent]         Download the revision from S3/GitHub
3. BeforeInstall          [user script]   Pre-install work (backups, etc.)
4. Install                [agent]         Copy the files section of appspec.yml
5. AfterInstall           [user script]   Post-install work (permissions, configuration)
6. ApplicationStart       [user script]   Start the app
7. ValidateService        [user script]   Health check

Blue/Green deployment (additional steps)
==========================
Blue instance: 1~7 + BeforeBlockTraffic → BlockTraffic → AfterBlockTraffic
Green instance: 1~7 + BeforeAllowTraffic → AllowTraffic → AfterAllowTraffic

DownloadBundle and Install are internal agent steps that cannot be customized by the user. If you don't know this, it's easy to misunderstand, thinking "I tried to fetch the files in advance in BeforeInstall but it didn't work." The bundle is packaged as zip/tar, and the agent unpacks it to /opt/codedeploy-agent/deployment-root/<deployment-group-id>/<deployment-id>/deployment-archive. BeforeInstall can access that unpacked directory, but it is the state just before files are copied to the destination (for example /var/www/html).

ValidateService plays the same role as Kubernetes' readiness/liveness probe.

version: 0.0
os: linux
 
files:
  - source: build/         # Path inside the unpacked directory
    destination: /var/www/myapp
  - source: nginx.conf
    destination: /etc/nginx/conf.d/myapp.conf
 
file_exists_behavior: OVERWRITE   # DISALLOW | OVERWRITE | RETAIN
                                  # DISALLOW: fail if an existing file is present (safe)
                                  # OVERWRITE: overwrite unconditionally
                                  # RETAIN: keep existing files, copy only new ones
 
permissions:
  - object: /var/www/myapp
    pattern: "**"
    owner: nginx
    group: nginx
    mode: 644
    type:
      - file
  - object: /var/www/myapp/bin
    pattern: "**"
    mode: 755
    type:
      - file
 
hooks:
  ApplicationStop:
    - location: scripts/stop.sh
      timeout: 60
      runas: root
  BeforeInstall:
    - location: scripts/backup.sh
      timeout: 30
  AfterInstall:
    - location: scripts/configure.sh
      timeout: 120
  ApplicationStart:
    - location: scripts/start.sh
      timeout: 60
  ValidateService:
    - location: scripts/healthcheck.sh
      timeout: 30

The ApplicationStop hook runs the script from the previous revision. That is, when deploying v1 → v2, the ApplicationStop script that was in v1 runs. Some people don't know this and say, "I added a new feature in ApplicationStop but it didn't run," but a script added in this deployment is first run as ApplicationStop at the next deployment. This is a subtle behavior that appears on the exam.

Deployment Configurations: Is AllAtOnce Really Dangerous?

CodeDeploy's "Deployment Configuration" determines how many instances are deployed to at the same time.

AllAtOnce's definition that "the deployment itself is a success even if only one succeeds" goes against intuition. The reason is CodeDeploy's philosophy that "deployment status" and "service availability" are separate. The deployment itself is finished, and if even one instance in the fleet received the new version, the intended job is complete. Whether the service is available as a result is a separate matter, which you must judge with CloudWatch Alarms. This becomes a trap in operations.

Netflix uses a "Rolling Red/Black" pattern in its own deployment tool Spinnaker, which is essentially the same as a pattern that verifies CodeDeploy's OneAtATime + ValidateService very strictly. Deploy the new version to one instance → let it receive traffic for 1-5 minutes while watching metrics → if healthy, move to the next instance, and if unhealthy, roll back immediately. You can produce almost the same effect with CodeDeploy's combination of OneAtATime + CloudWatch Alarm + Auto Rollback.

Blue/Green: The Luxury of Spinning Up Two Whole Environments

An In-Place deployment replaces the files on the same instance. It is fast and costs nothing extra, but rollback is hard because the previous files are already gone. Blue/Green spins up a whole new ASG, deploys v2 to the new instances, and swaps the ALB Target Group to the new ASG. If something goes wrong, you just swap back to the original ASG and you're done.

[ALB] ──── Production Listener (port 80)
   │           │
   │           ├── Target Group "blue"  ← ASG-blue (v1, 3 instances)
   │           │
   │           └── Target Group "green" ← ASG-green (v2, 3 instances) [new]
   │
   └── [As the deployment proceeds, the Listener Rule switches to green]

The trade-offs of Blue/Green:

"What happens to the original (Blue) ASG after a Blue/Green deployment?" appears on the exam. There are two options: ① terminate immediately (Terminate original instances immediately) or ② terminate after a specified time (Wait N minutes then terminate). The latter acts as a safety net: keeping Blue alive briefly even after the swap allows a fast rollback. It is usually set somewhere between 5 minutes and 1 hour.

Lambda Deployment: Slicing Traffic with the Alias routing-config

Lambda is a completely different mechanism from EC2. It doesn't copy files anywhere; it publishes a new version of the function and adjusts the traffic ratio with the alias routing-config.

# Lambda alias traffic split (Boto3)
import boto3
lambda_client = boto3.client('lambda')
 
# Configure alias 'live' so v1 receives 90% and v2 receives 10%
lambda_client.update_alias(
    FunctionName='my-function',
    Name='live',
    FunctionVersion='1',                # Primary version
    RoutingConfig={
        'AdditionalVersionWeights': {
            '2': 0.1                    # v2 gets 10%
        }
    }
)

CodeDeploy's Lambda deployment automates this update_alias call on a predefined schedule. There are 9 predefined configurations.

The difference between Canary and Linear is the heart of the exam. Canary is a two-step shift: "hold 10% for N minutes and watch, then go to 90% all at once." You monitor for problems at the first 10%, then go to the whole with one big decision. Linear increases evenly every N minutes: "10%, 20%, 30%, ...". Canary is fast but carries the risk of concentrating on one side, whereas Linear is slow and gradual. Linear suits cases that need A/B testing, and Canary suits cases that need quick validation followed by a full deployment.

# Lambda appspec.yml
version: 0.0
Resources:
  - MyLambdaFunction:
      Type: AWS::Lambda::Function
      Properties:
        Name: my-function
        Alias: live
        CurrentVersion: 1
        TargetVersion: 2
 
Hooks:
  - BeforeAllowTraffic: pre-traffic-validation-fn      # Validate before receiving traffic
  - AfterAllowTraffic: post-traffic-validation-fn      # Validate after the shift completes

The Lambda function called from a hook must call the PutLifecycleEventHookExecutionStatus API at the end to report Succeeded/Failed. If it doesn't, the deployment stays stuck until the timeout (default 1 hour).

def lambda_handler(event, context):
    # The event contains deploymentId and lifecycleEventHookExecutionId
    deployment_id = event['DeploymentId']
    hook_execution_id = event['LifecycleEventHookExecutionId']
 
    # Validation logic (smoke test, contract test, etc.)
    success = run_smoke_test()
 
    codedeploy = boto3.client('codedeploy')
    codedeploy.put_lifecycle_event_hook_execution_status(
        deploymentId=deployment_id,
        lifecycleEventHookExecutionId=hook_execution_id,
        status='Succeeded' if success else 'Failed'
    )

If the BeforeAllowTraffic hook fails, it rolls back immediately with 0% of traffic on the new version. If the AfterAllowTraffic hook fails, it automatically rolls back to the previous version even though the shift has already finished. In other words, both hooks become automatic rollback triggers. If "behavior when validation fails during a Lambda deployment" appears on the exam, look at these two hooks and automatic rollback together.

Automatic Rollback Based on CloudWatch Alarms

If the error rate spikes during or right after a deployment, you want to roll back automatically. CodeDeploy is integrated with CloudWatch Alarms.

# DeploymentGroup settings (CloudFormation excerpt)
DeploymentGroup:
  AutoRollbackConfiguration:
    Enabled: true
    Events:
      - DEPLOYMENT_FAILURE        # The deployment itself failed
      - DEPLOYMENT_STOP_ON_ALARM  # Stopped because an alarm fired
      - DEPLOYMENT_STOP_ON_REQUEST # Manual stop by the user
  AlarmConfiguration:
    Enabled: true
    Alarms:
      - Name: lambda-error-rate-high
      - Name: lambda-duration-high
    IgnorePollAlarmFailure: false

If IgnorePollAlarmFailure is false, the deployment fail-stops when the alarm status can't be retrieved from CloudWatch. It is conservative from a safety standpoint, but deployments get blocked during a temporary CloudWatch outage. If set to true, the deployment simply proceeds when the alarm lookup fails, which favors availability. Neither is the right answer; it depends on the operational policy.

ECS Blue/Green: Swapping the Whole Target Group

With CodeDeploy, ECS supports only Blue/Green. The principle is similar to EC2 Blue/Green, but the unit is not an instance but the Task definition revision.

[ALB Production Listener:80]
       │
       ├── Target Group "blue"  ← Blue Tasks of the ECS Service (revision N)
       │
       └── Target Group "green" ← Green Tasks of the ECS Service (revision N+1) [new]

[ALB Test Listener:8080]    ← A separate Listener for validating Green during the deployment
       │
       └── Target Group "green"

The deployment flow:

1. CodeDeploy launches tasks (Green) in the ECS service with the new task definition
2. Once all Green tasks are healthy, the Production listener is switched to Green
3. (Optional) Keep the Blue tasks alive for a specified time (a safety net for fast rollback)
4. After the time elapses, terminate the Blue tasks

To use ECS Blue/Green you need four things.

• 2 ALB Target Groups
• A Production Listener and (optionally) a Test Listener configured
• The ECS service's deploymentController.type set to CODE_DEPLOY
• An appspec.json file specifying the task definition + load balancer info

If even one is missing, the deployment fails to start at all. If "prerequisites for ECS Blue/Green" appears on the exam, all four of these are the answer.

{
  "version": 0.0,
  "Resources": [
    {
      "TargetService": {
        "Type": "AWS::ECS::Service",
        "Properties": {
          "TaskDefinition": "arn:aws:ecs:...:task-definition/myapp:42",
          "LoadBalancerInfo": {
            "ContainerName": "app",
            "ContainerPort": 80
          },
          "PlatformVersion": "LATEST"
        }
      }
    }
  ],
  "Hooks": [
    { "BeforeInstall": "pre-validation-fn" },
    { "AfterInstall": "post-task-up-fn" },
    { "AfterAllowTestTraffic": "smoke-test-fn" },
    { "BeforeAllowTraffic": "final-check-fn" },
    { "AfterAllowTraffic": "post-deploy-fn" }
  ]
}

Comparison with Other Deployment Tools

CodeDeploy Agent Operations Tips

# Install on Amazon Linux 2
sudo yum install -y ruby wget
wget https://aws-codedeploy-${REGION}.s3.${REGION}.amazonaws.com/latest/install
chmod +x ./install
sudo ./install auto
 
# Manage with systemd
sudo systemctl enable codedeploy-agent
sudo systemctl start codedeploy-agent
sudo systemctl status codedeploy-agent
 
# Log locations (essential for debugging)
tail -f /var/log/aws/codedeploy-agent/codedeploy-agent.log
tail -f /opt/codedeploy-agent/deployment-root/deployment-logs/codedeploy-agent-deployments.log
 
# All hook logs for a specific deployment
ls /opt/codedeploy-agent/deployment-root/<deployment-group-id>/<deployment-id>/logs/

The agent is written in Ruby, so Ruby must be installed for it to work. If a minimal container image has no Ruby, the agent installation fails.

Summary

The essence of CodeDeploy is "breaking the risky moment of deployment into several small steps and embedding validation and rollback mechanisms in each step." It implements the same philosophy in different ways: with lifecycle hooks on EC2, with alias weights on Lambda, and with a task definition swap on ECS.

In the next post we look at the orchestrator that flows all these steps automatically: CodePipeline. We cover how to manage the entire flow, from detecting source changes through build, test, approval, and deployment, as code, and we touch on multi-Region and multi-account pipeline patterns.