Day 3 - AWS Config: Closed-Loop Control Principles for State Recording, Drift Detection, and Automated Remediation

The day after infrastructure is defined as code and deployed, someone in the console changes a line in a security group by hand. That one line becomes an incident days later. The most persistent problem in cloud operations is this: "The declared state (desired state) and actual state (actual state) diverge over time" — this is drift. AWS Config addresses this problem directly: "Record every change in every resource without exception, continuously evaluate whether it complies with policy, and when it deviates, automatically restore it." From the console, Config looks like a "service that shows resource inventory and compliance scores," but beneath that lie the concepts of feedback control from control theory, the history of configuration management, and declarative policy evaluation. Today we examine how Config records every change as Configuration Items, how Rules evaluate compliance, how Remediation uses SSM Automation to auto-correct drift, and where the costs and pitfalls lie.

In the DOP exam, Config appears as the core of "compliance evaluation + automated drift remediation," with scenarios like "to instantly correct specific policy violations without human intervention," "to deploy multiple standards at once across the organization," and "to view compliance across multiple accounts and regions from one place." Distinguishing the 3 Rule types (Managed/Custom Lambda/Custom Policy), connecting Remediation to SSM Automation, and understanding Conformance Packs and Aggregators reveals the answers.

Configuration Management and Closed-Loop Control — Config's Philosophical Roots

Config's concept is not new. Configuration Management originated as a discipline in the 1950s U.S. defense and aerospace industries: "track the state of all components in complex systems and maintain control so that they never deviate from the approved baseline." Transitioning to software, tools like CFEngine (1993), Puppet (2005), Chef (2009), and Ansible (2012) implemented this by "converging and maintaining servers toward their declared state."

The common principle underlying these tools is closed-loop control. In control theory, a closed-loop (feedback control) system ①compares the goal state (setpoint) with ②the measured actual state, ③calculates the error, and ④manipulates in the direction that reduces that error. A thermostat is the classic example — it compares the set temperature with actual temperature and turns the heater on and off.

AWS Config applies this closed loop to infrastructure precisely. The goal state is the Config Rule (policy), the measurement is the Configuration Recorder (state recording), the error judgment is Rule evaluation (COMPLIANT/NON_COMPLIANT), and the manipulation is Remediation (automated correction). When drift occurs, an error is detected and Remediation restores the actual state to the goal — just as a thermostat restores a room to its set temperature.

💡 Related Theory: Configuration management tools broadly share convergent and idempotent semantics. Idempotency is the property that "applying the same operation multiple times produces the same result as applying it once" (mathematically, f(f(x)) = f(x)), as in Ansible's "if already at goal state, do nothing." Config's Remediation must also be idempotent — executing the SSM document that corrects NON_COMPLIANT states multiple times must have no side effects. Another key distinction is imperative ("execute these commands") vs. declarative ("reach this state"). Config is strictly declarative — it declares only the goal "S3 must be encrypted," leaving how to reach it to Remediation. Declarativeness's advantage is that it converges toward the goal regardless of the current state.

Configuration Item — Point-in-Time Snapshots of Every Change

The foundation of Config is the Configuration Item (CI). A CI is a snapshot capturing a resource's state at one point in time — including attributes, relationships (which SG, subnet, and EBS this EC2 is attached to), metadata, and even the CloudTrail event that triggered the change. Each time a resource changes, a new CI is created, forming a Configuration History. This history is stored in S3 and can trigger SNS notifications on change.

CIs are powerful because of the time axis. "What state was this security group in 6 months ago?" and "what changed just before this modification?" become answerable. This is the foundation for incident forensics and change auditing.

🔍 Deeper: Config recording relationships between resources is more important than just recording attributes. A CI captures graph structure: "EC2 i-xxx is attached to SG sg-yyy, located in subnet subnet-zzz, has EBS vol-www attached." This constructs a resource dependency graph rather than a simple list. Queries like "what resources are affected if we delete this SG?" and "what is in this subnet?" become possible. This is Config's cloud-native implementation of a CMDB (Configuration Management Database) — the central repository of IT assets and relationships defined by ITIL. This is why Advanced Query (coming next) can query this graph with SQL.

⚠️ Pitfall: Don't confuse Config with CloudTrail. CloudTrail records "who called what API when" (behavior/audit log), while Config records "what state did the resource end up in as a result" (state/configuration). "Who changed this SG?" is CloudTrail; "what rules did this SG have then and now?" is Config. They're complementary — CIs link to the CloudTrail events that caused the change, giving you the complete picture: "who → did what → to what state." In exams: "track resource state changes and compliance" is Config; "audit API calls and users" is CloudTrail.

Config Rule — Three Types of Policy Evaluation

Config Rules evaluate whether resources comply with policy. There are three types, and this distinction appears directly in exams.

Representative Managed Rules: s3-bucket-public-read-prohibited, encrypted-volumes, iam-password-policy, restricted-ssh, root-account-mfa-enabled, rds-storage-encrypted.

Evaluation triggers are two-fold. A Configuration change trigger evaluates every time a resource changes (near-real-time), while a Periodic trigger evaluates on a fixed schedule (e.g., 24 hours). Rules depending on external state should use periodic evaluation, while rules that need to catch violations immediately should use change triggers.

🔍 Deeper: Why Custom Policy Rule's Guard DSL matters. CloudFormation Guard is a domain-specific language expressing policy declaratively (policy-as-code) like Resources.*[ Type == "AWS::S3::Bucket" ] { Properties.BucketEncryption EXISTS } saying "resources of this type must have this property." It's in the lineage of Kubernetes's OPA (Open Policy Agent)/Rego, HashiCorp's Sentinel, and other policy engines. The key value is "instead of writing evaluation logic in Lambda code (imperative), express policy declaratively so humans can read, verify, and version-control it." Guard rules can be reused in both pre-deployment (checking templates with cfn-guard validate in CI/CD) and post-deployment (evaluating real resources as Config Custom Policy Rules), unifying "shift-left" (blocking before deploy) and "runtime evaluation" under one policy language.

rule s3_bucket_must_be_encrypted {
  Resources.*[ Type == "AWS::S3::Bucket" ] {
    Properties.BucketEncryption EXISTS
  }
}

Conformance Pack — Bundled Policy Deployment

Deploying rules one-by-one isn't realistic for dozens of rules across hundreds of accounts. A Conformance Pack bundles multiple Config Rules + Remediation into a single YAML and deploys them all at once. Pre-defined packs are provided (PCI DSS, NIST 800-53, FedRAMP, HIPAA, AWS Well-Architected, etc.), and custom YAMLs are also supported.

Resources:
  S3PublicProhibitedRead:
    Type: AWS::Config::ConfigRule
    Properties:
      ConfigRuleName: s3-public-read
      Source:
        Owner: AWS
        SourceIdentifier: S3_BUCKET_PUBLIC_READ_PROHIBITED
      Scope:
        ComplianceResourceTypes: [AWS::S3::Bucket]

aws configservice put-conformance-pack --conformance-pack-name security-baseline \
  --template-s3-uri s3://conformance-packs/security-baseline.yaml

🔍 Deeper: Conformance Pack internally uses CloudFormation StackSets for org-wide deployment. That is, "define a rule bundle, then StackSets pushes it identically to each account and region." So Conformance Pack inherits StackSets' deployment model (parallel/sequential, failure tolerance thresholds). To clarify the relationship: StackSets is a general-purpose tool for multi-account deployment of any CloudFormation resources, while Conformance Pack is a specialized abstraction on top for "Config Rule + Remediation bundles." In exams: "deploy PCI/NIST compliance bundles org-wide" is Conformance Pack; "deploy arbitrary infrastructure resources multi-account" is StackSets.

Auto-Remediation — The Manipulation Phase That Closes the Loop

Evaluation alone doesn't close the loop. When NON_COMPLIANT is found, it must actually be fixed. Config's Remediation links a Rule to an SSM Automation Document, executing that document when a violation is found to correct the state.

aws configservice put-remediation-configurations \
  --remediation-configurations '[{
    "ConfigRuleName":"s3-bucket-public-read-prohibited",
    "TargetType":"SSM_DOCUMENT",
    "TargetId":"AWS-DisableS3BucketPublicReadWrite",
    "Parameters":{
      "AutomationAssumeRole":{"StaticValue":{"Values":["arn:aws:iam::...:role/RemediationRole"]}},
      "S3BucketName":{"ResourceValue":{"Value":"RESOURCE_ID"}}
    },
    "Automatic":true,
    "MaximumAutomaticAttempts":5,
    "RetryAttemptSeconds":60
  }]'

If Automatic: true, violations are corrected immediately upon detection; if false, an operator triggers correction manually from the console. MaximumAutomaticAttempts and RetryAttemptSeconds control retry behavior.

⚠️ Pitfall: Auto-Remediation has dangerous scenarios — fix-violate loops (flapping). If some external process keeps returning a resource to NON_COMPLIANT state (e.g., another automation opens an SG, Config closes it, another opens it again...), Config attempts endless corrections, causing API throttling, cost explosion, and noise. MaximumAutomaticAttempts is the safety valve that stops this loop. Also, Remediation can be destructive — "force S3 buckets private" can break legitimate public website buckets. So production often puts exception tags on important resources or makes destructive remediations Automatic:false requiring human approval. In exams, "auto-remediation repeats endlessly" usually answers to "external process keeps violating + MaximumAutomaticAttempts not set."

📚 Case Study: A famous incident illustrating drift and bad remediation is GitLab's 2017 database deletion. An engineer manually deleting a production database directory while responding to a replication delay found that all five backup mechanisms failed, losing ~6 hours of data. The core lesson: "manual intervention is the enemy of configuration management, and safety nets (backups, validation) must be verified to actually work at all times." This is Config's value — it constantly evaluates that safety nets themselves (backup settings, encryption, replication) are enabled (e.g., db-instance-backup-enabled rule), catching NON_COMPLIANT immediately and auto-correcting. The philosophy is "don't trust that safety nets are on; prove they're on at all times" — this is the spirit of closed-loop compliance.

Aggregator — Unified Multi-Account, Multi-Region View

Since Config is per-account and per-region, viewing organization-wide compliance requires an Aggregator. Create an Aggregator in the Audit account with Organization as source, and all member accounts' Config data across all regions becomes unified.

aws configservice put-configuration-aggregator --configuration-aggregator-name org-aggregator \
  --organization-aggregation-source RoleArn=arn:aws:iam::...:role/AWSConfigOrgRole,AllAwsRegions=true

Advanced Queries — SQL Queries Over Inventory

Advanced Query queries Config data with SQL DSL. Similar syntax to CloudWatch Logs Insights, and cross-account queries are possible via Aggregator.

SELECT
  configuration.targetResource.resourceType,
  COUNT(*) as count
WHERE configuration.complianceType = 'NON_COMPLIANT'
GROUP BY configuration.targetResource.resourceType

This is the interface that makes the resource dependency graph (CMDB) queryable — "how many unencrypted EBS volumes are in which accounts" is answered in one query.

Config + EventBridge — Routing Compliance Changes

Config emits events to EventBridge when compliance state changes. These can be routed to Slack alerts, Lambda, or SSM (a more flexible path than Remediation alone).

{
  "source": ["aws.config"],
  "detail-type": ["Config Rules Compliance Change"],
  "detail": {"newEvaluationResult": {"complianceType": ["NON_COMPLIANT"]}}
}

Cost Control — recordingMode

Config charges per resource change (CI creation). Frequently-changing resources (e.g., EC2 constantly spawned and terminated by Auto Scaling, frequently-refreshed metadata) drive up costs. recordingMode controls whether to record all resources, only certain types, or bundle high-frequency resources by day.

💡 Related Theory: This cost structure exemplifies observability's fundamental tradeoff — "the more you record, the better you see, but the more it costs" — the same thinking as sampling in distributed tracing (keeping only some traces). With Config, you must choose between "record every change flawlessly (full audit trail, high cost)" and "record only important resources (low cost, blind spots)." The practical pattern is: record must-comply resources (IAM, S3, SG) completely, and exclude noisy, regulation-agnostic resources. The principle "observability isn't free" sits at the core of Config's cost design.

Config vs Security Hub Standards

As seen in Day 2, many Security Hub Standards controls run Config Rules internally. Config is the evaluation engine, and Security Hub is the layer above that consolidates and visualizes results in ASFF.

🎯 Scenario: "Organization-wide (OU): ① all S3 buckets in all new/existing accounts must not be public and must be encrypted, auto-corrected instantly; ② EBS must be encrypted ③ view non-compliant resources across all accounts/regions in Audit account; ④ custom policy (tags mandatory) without Lambda; ⑤ protect against endless auto-remediation loops." → ① S3 public/encryption Managed Rules bundled in Conformance Pack deployed to OU via StackSets + SSM Automation Remediation (Automatic:true) on each rule; ② encrypted-volumes Managed Rule + Remediation; ③ Audit account Organization Aggregator (AllAwsRegions) + Advanced Query; ④ tag-mandatory policy as Custom Policy Rule (Guard DSL); ⑤ Remediation with MaximumAutomaticAttempts/RetryAttemptSeconds + exception tags on justified resources. Compliance changes routed via EventBridge to Slack alerts.

Summary

Today we saw four key points. First, Config applies closed-loop control to infrastructure — goal (Rule), measurement (Recorder), error judgment (evaluation), and manipulation (Remediation) following the thermostat model, standing in the lineage of declarative, idempotent configuration management (Puppet/Chef/Ansible). Second, Configuration Item records resource point-in-time state and relationship graph (CMDB). This complements CloudTrail's API-call focus by handling resource state. Third, Rules come in 3 types: Managed/Custom Lambda/Custom Policy (Guard DSL, OPA/Sentinel lineage); Conformance Packs bundle via StackSets; Remediation closes the loop via SSM Automation, preventing flapping and destructive fixes with MaximumAutomaticAttempts and exception tags. Fourth, Aggregator unifies multi-account/region, Advanced Query enables SQL, recordingMode manages observability cost tradeoffs, and Security Hub Standards leverages Config as the underlying evaluation engine.

Next we'll extend compliance into automated audit evidence collection, data classification, and vulnerability scanning via Audit Manager, Macie, and Inspector.

📌 Today's Summary

Today's key points were four-fold. First, Config applies closed-loop (feedback) control to infrastructure — goal (Rule), measurement (Recorder), error judgment (evaluation), and manipulation (Remediation) follow the thermostat model, standing in the lineage of declarative, idempotent configuration management. Second, Configuration Item records both point-in-time resource state and relationship graph (CMDB), complementing CloudTrail's audit of "API calls" with management of "resource state." Third, Rules come in 3 types: Managed/Custom Lambda/Custom Policy (Guard DSL, OPA/Sentinel lineage); Conformance Pack bundles via StackSets; Remediation closes the loop via SSM Automation while preventing flapping and destructive fixes with MaximumAutomaticAttempts and exception tags. Fourth, Aggregator unifies multi-account/region viewing, Advanced Query enables SQL querying, recordingMode controls observability cost tradeoffs, and Security Hub Standards uses Config as the underlying evaluation engine.