Day 2 - SSM Automation: Operational Procedures as Code, and People Inside the Workflow

The deepest debt in an operations organization lives not in code but in people's heads. "When restarting the production DB, first check the Read Replica lag, see whether the backup has finished, get approval from two senior engineers, drain the traffic, restart it, and once the health check passes, put the traffic back." This procedure exists only in a wiki page or in a veteran's experience. When that veteran goes on vacation or leaves the company, the same procedure gets reproduced at 3 a.m. by an on-call engineer's trembling hands, and if one step is skipped, an incident happens. A runbook is this procedure written down in prose, and Runbook-as-Code is that procedure turned into executable code. SSM Automation is the tool that implements this on AWS — you write the operational procedure as a declarative workflow and weave AWS API calls, scripts, conditional branches, and human approvals into it as steps.

Today we don't look at SSM Automation merely as a "remediation executor"; we dig into the workflow model underneath. We look at which workflow-engine lineage the Automation Document belongs to, why human-in-the-loop steps such as aws:approve resolve the paradox of automation, how Step Functions and Automation differ, and which fundamental problems of workflow engines idempotency and output passing address. In the DOP exam, SSM Automation is the flagship service for Operational Excellence and incident automation, and it is tested as "how do you codify a procedure," "how do you insert a human gate into an automated response," and "how do you apply the same remediation to hundreds of accounts at once."

The History of the Runbook — From Paper on the Operations Room Wall to Code

The word "runbook" comes from mainframe-era data centers. The binder in which operators wrote down the procedures for overnight batch jobs, incident response, and system restarts was literally a "run book." When the computer stopped, the operator opened that book and followed the steps. As this paper procedure was digitized it became wikis and checklists, and together with the DevOps movement it evolved into the concept of the executable runbook — the idea of turning a procedure from prose that people read into code that machines execute.

This evolution has stages. First paper (human-executed), then a collection of scripts (semi-automated), then a workflow engine (fully orchestrated). SSM Automation is the last stage: each step of the procedure is written as declarative YAML/JSON, and SSM manages the state transitions. People declare only "what to do," and the engine takes care of "how to execute, retry, and branch."

[Paper binder in the ops room] → [Wiki checklist] → [Individual scripts] → [SSM Automation Document]
   human-executed                  semi-manual         scripted              fully orchestrated
   (depends on veterans)           (copy-paste runs)   (glue code)           (declarative workflow + audit)

💡 Related theory: Runbook-as-Code is the implementation of "operations as code," a core principle of Operational Excellence. The Operational Excellence pillar of the AWS Well-Architected Framework says, "Defining operational procedures as code enables consistency, reproducibility, version control, and testing." It applies the same philosophy as treating infrastructure as code (IaC) to operational procedures. The key benefits: when a procedure is code, (1) it is version-controlled and reviewed in Git, (2) it is identical no matter who runs it (removing dependence on veterans), (3) every execution is audited in CloudTrail, and (4) when you improve the procedure after an incident, the next incident automatically benefits. This connects to the SRE principle that "the surest way to reduce human error is to take humans out of the procedure."

The Automation Document — Anatomy of a Declarative Workflow

There are several kinds of SSM Document, and among them Automation (schemaVersion: '0.3') is the one for workflows.

The skeleton of an Automation Document is parameters + main steps (mainSteps).

schemaVersion: '0.3'
description: Auto-remediate unencrypted S3 buckets
assumeRole: '{{AutomationAssumeRole}}'   # The IAM role this workflow will assume
parameters:
  BucketName: {type: String}
  AutomationAssumeRole: {type: String}
mainSteps:
  - name: CheckEncryption
    action: aws:executeAwsApi
    inputs:
      Service: s3
      Api: GetBucketEncryption
      Bucket: '{{BucketName}}'
    onFailure: 'step:EnableEncryption'   # Branch on failure
    isCritical: false
  - name: EnableEncryption
    action: aws:executeAwsApi
    inputs:
      Service: s3
      Api: PutBucketEncryption
      Bucket: '{{BucketName}}'
      ServerSideEncryptionConfiguration:
        Rules: [{ApplyServerSideEncryptionByDefault: {SSEAlgorithm: AES256}}]

assumeRole is the key — Automation operates with the permissions of the IAM role specified in the document, not the permissions of the person who ran it. Thanks to this, privilege delegation becomes possible: "an operator without permissions can perform limited tasks through an approved runbook."

The assumeRole of an Automation Document is an important security pattern: repositioning the permission boundary. Instead of giving operators s3:PutBucketEncryption directly, you put that permission in the runbook's role and give operators only ssm:StartAutomationExecution (restricted to specific documents). Then operators can use that permission only through "this vetted procedure" and cannot wield it arbitrarily. This implements the principle of least privilege at the level of a procedure — the same idea as sudo allowing privilege escalation for "specific commands only." On the exam, the answer to "allow only controlled tasks without giving operators dangerous permissions directly" is often this pattern.

Kinds of Step Actions — The Primitive Operators of the Workflow

Automation's expressiveness comes from step actions. Each step has one action type.

These steps can pass outputs to the next step.

- name: GetSnapshot
  action: aws:executeAwsApi
  outputs:
    - {Name: SnapshotId, Selector: $.SnapshotId, Type: String}
  inputs: {Service: ec2, Api: CreateSnapshot, VolumeId: '{{VolId}}'}
- name: TagSnapshot
  action: aws:executeAwsApi
  inputs:
    Service: ec2
    Api: CreateTags
    Resources: ['{{GetSnapshot.SnapshotId}}']   # Reference the previous step's output

Output passing between steps ({{Step.Output}}) and branching (aws:branch) make Automation not a simple script but a stateful workflow. This model connects to DAG (directed acyclic graph) execution, the classic of workflow engines — a graph in which each step is a node and output dependencies are edges. Apache Airflow's DAGs, AWS Step Functions' state machines, and GitHub Actions' job dependencies all share the same lineage. However, Automation does not provide branching, parallelism, and Map as rich as Step Functions — so when complex branching is needed, it delegates to Step Functions with aws:executeStateMachine. The boundary: operational procedures that are linear to lightly branching go to Automation; complex state transitions, parallelism, and long-running execution go to Step Functions.

aws:approve — The Human Gate That Resolves the Paradox of Automation

The paradox of full automation is that the more dangerous the task, the scarier it is to automate. If you leave "terminate the production DB" to unattended automation, a single false positive kills a perfectly healthy DB. But if a person does everything by hand from start to finish, the benefit of automation disappears. aws:approve is the answer in between — it stops the workflow right before the dangerous step and waits for human approval.

- name: ApproveTermination
  action: aws:approve
  timeoutSeconds: 3600
  inputs:
    NotificationArn: arn:aws:sns:...:OncallTopic
    Message: 'Terminate prod instance {{InstanceId}}? GuardDuty critical finding.'
    MinRequiredApprovals: 2
    Approvers:
      - 'arn:aws:iam::...:role/SeniorOps'
      - 'arn:aws:iam::...:user/secops-lead'

MinRequiredApprovals: 2 is multi-party authorization — it proceeds only with approval from two or more people, not one. It is the same idea as the "two-person rule" for nuclear launch codes.

aws:approve is the implementation of human-in-the-loop (HITL) automation. Between fully autonomous and fully manual, it is a model in which the system automates most of the work but inserts human judgment only at decisive, irreversible points. The important distinction here is reversibility — like Jeff Bezos's "Type 1 (irreversible) vs. Type 2 (reversible) decisions" framing, tasks that can be undone (adding a tag) are automated quickly, and tasks that cannot be undone (terminating an instance, deleting data) get a human approval gate. The multi-party approval of MinRequiredApprovals: 2 is the security separation of duties and the two-person rule, preventing one person's mistake or malice from carrying out a destructive action alone. This is why compliance frameworks such as SOX and PCI-DSS require separation of duties.

⚠️ Trap: You must always set a timeout on aws:approve. If an approval step has no timeout, then when the approver misses the notification or is away, the workflow is stuck indefinitely — the automation freezes forever in the "waiting for approval" state in the middle of an incident. Set timeoutSeconds, and on timeout use onFailure to branch to escalation (calling the next approval group) or to a safe default action (do nothing and hand off to a person). One more thing: verify that the approval notification ARN (SNS) actually reaches people — if the topic subscription is broken, the approval request vanishes into thin air.

AWS Managed Runbooks — Don't Reinvent the Wheel

AWS provides hundreds of prebuilt Automation Documents. For common operational tasks, the standard practice is to look for a managed document before writing your own.

The AWSConfigRemediation-* family in particular ties directly into automatically remediating AWS Config rule violations — when Config detects "this bucket is not encrypted," the corresponding remediation document fixes it automatically.

A fintech company was told in a compliance audit that "publicly accessible S3 buckets" kept appearing periodically. Developers repeatedly left buckets open to the public by mistake. They detected it with the AWS Config rule s3-bucket-public-read-prohibited and connected the managed runbook AWS-DisablePublicAccessForS3Bucket as automatic remediation on violations. But they missed one thing at first — when they applied immediate remediation without aws:approve, it also closed the static website buckets that were meant to be public, and the service went down. The lesson: automatic remediation is powerful, but if you apply it across the board without a tag-based filter or approval gate that distinguishes exceptions (intentional settings), it breaks healthy resources. Afterward they narrowed the scope so that buckets tagged Exempt=true were excluded from remediation.

EventBridge → Automation — From Detection to Response

If the EventBridge we saw yesterday detects "what happened," SSM Automation executes "so what will we do about it." Connecting these two is the core pattern of automated incident response.

// EventBridge Rule: GuardDuty high-severity detection
{
  "source": ["aws.guardduty"],
  "detail-type": ["GuardDuty Finding"],
  "detail": {
    "severity": [{"numeric": [">=", 7]}],
    "type": [{"prefix": "UnauthorizedAccess:EC2/"}]
  }
}

If you specify an SSM Automation document as the target, a GuardDuty detection immediately triggers the isolation workflow.

{
  "DocumentName": "AWS-IsolateEC2Instance",
  "Parameters": {"InstanceId": ["$.detail.resource.instanceDetails.instanceId"]}
}

🎯 Scenario: GuardDuty detected Bitcoin mining traffic (CryptoCurrency:EC2/BitcoinTool) from an EC2 instance with severity 8. Target flow: (1) an EventBridge Rule catches findings with severity>=7 and triggers SSM Automation → (2) Automation moves the instance to an isolation security group (sg-quarantine) with aws:executeAwsApi (blocking inbound and outbound) → (3) creates a forensic snapshot with aws:createImage → (4) sends a Slack notification and Jira ticket with aws:invokeLambdaFunction → (5) waits for senior approval with aws:approve → (6) once approved, terminates the instance with aws:executeAwsApi. The key point is that isolation, evidence preservation, and notification run immediately and unattended (reversible, non-destructive), while termination (irreversible, destructive) comes after human approval. The reason to isolate before terminating is MTTR — contain the threat immediately to stop damage from spreading, and do the analysis and termination slowly afterward.

Multi-Account Automation — One Procedure Across the Whole Organization

As you scale up, you have to apply the same remediation to hundreds of accounts. The TargetLocations parameter rolls one Automation execution out to multiple accounts and Regions at once.

aws ssm start-automation-execution \
  --document-name AWS-DisablePublicAccessForS3Bucket \
  --target-locations '[{
    "Accounts": ["ou-abcd-12345678"],
    "Regions": ["us-east-1", "ap-northeast-2"],
    "TargetLocationMaxConcurrency": "10",
    "TargetLocationMaxErrors": "5"
  }]'

MaxConcurrency (the number of concurrent executions) and MaxErrors (the number of tolerated errors) are the key safeguards.

MaxConcurrency/MaxErrors provide blast radius control for large-scale automation. If you push a change to all accounts at once, a bug breaks the whole organization simultaneously. Limiting concurrency makes the change spread gradually, and when the error threshold (MaxErrors) is exceeded it stops automatically to contain the damage. This is exactly the same idea as the canary/rolling deployment strategies — start small, expand while checking health, and stop when problems appear. The first principle of safe operations, "every change should be gradual and reversible," applies to automation executions as well.

Automation vs. Step Functions vs. Lambda — The Boundaries of Tool Choice

These three tools are often compared. You need clear boundaries so you don't get confused on the exam.

The boundaries among the three are a trade-off between expressiveness and simplicity. Lambda is the simplest, but you have to write orchestration (retries, branching, state) yourself in code. Step Functions has the highest expressiveness (parallelism, Map, Choice, long waits) but comes with a learning curve and the cost of writing ASL (Amazon States Language). Automation sits in between and is specialized for operational tasks — AWS API calls, instance commands, managed runbooks, and human approval are first-class citizens, so it is the most natural for writing operational procedures. Anti-patterns: building a complex business workflow (order → payment → shipping) in Automation, or over-engineering a simple operational remediation with Step Functions. The tool should match the shape of the problem.

Summary

There are five pictures from today. First, Runbook-as-Code evolves a procedure that began as a paper binder in the ops room into executable code, implementing the Operational Excellence principle of "operations as code." Second, the assumeRole of an Automation Document repositions the permission boundary at the procedure level, enabling controlled privilege delegation. Third, aws:approve resolves the paradox of automation through human-in-the-loop, placing a human gate according to reversibility and implementing separation of duties through multi-party approval (the two-person rule). Fourth, EventBridge→Automation connects detection to response, and the key is the order: isolation (reversible) immediately, termination (irreversible) after approval. Fifth, MaxConcurrency/MaxErrors in Multi-Account is blast radius control, following the same principle of gradual, reversible change as canary/rolling.

In the next article, we take a deep look at the Lambda auto-healing pattern, in which the system heals itself without even human approval, and at the safety nets that stop it from running away.

📌 Today's Summary

There are five key points from today. First, Runbook-as-Code is the implementation of "operations as code," evolved from the paper binder in a mainframe operations room into executable code, giving consistency, reproducibility, auditing, and removal of dependence on veterans. Second, the assumeRole of an Automation Document repositions the permission boundary at the procedure level, implementing least-privilege delegation in which operators use dangerous permissions only through a vetted runbook instead of being given them directly. Third, aws:approve resolves the paradox of automation through human-in-the-loop, placing a human gate only on irreversible tasks according to reversibility (Type 1/Type 2 decisions) and implementing separation of duties with MinRequiredApprovals (the two-person rule), with a timeout being mandatory. Fourth, EventBridge→Automation connects detection (what happened) to response (what to do), the key being the order — isolation and evidence preservation immediately and automatically, termination after approval — and managed runbooks mean you don't reinvent the wheel. Fifth, TargetLocations and MaxConcurrency/MaxErrors in Multi-Account provide blast radius control, applying the gradual, reversible change principle of canary/rolling to automation executions.