Day 4 - Audit Manager, Macie, Inspector: Evidence Automation, Data Classification, Vulnerability Scanning Principles

The last piece of the security automation stack is offloading to machines "the three things humans most hate doing." Compliance audit season's manual screenshot and log collection, finding buried SSNs and card numbers in petabyte-scale data lakes, and cross-referencing thousands of incoming CVEs against thousands of workloads. Audit Manager, Macie, and Inspector handle these three respectively. From the console they look like "plug-and-play compliance and security tools," but beneath lies the philosophy of continuous compliance, machine-learning-based data classification with pattern matching, the history of CVE/CVSS vulnerability standardization, and proving least privilege with IAM data. Today we examine what each does internally and how IAM Access Analyzer, Firewall Manager, and CloudTrail Lake complete the overall security automation stack.

In the DOP exam, this domain appears as "automated audit evidence collection + sensitive data discovery + vulnerability management + external exposure detection," with scenarios like "collect SOC2 audit evidence continuously," "auto-discover PII in S3," "scan container images for dependency CVEs at push," and "find resources exposed to external accounts." Distinguishing each tool's role boundaries reveals the answers.

Continuous Compliance — Transforming Audit from Event to State

Traditional compliance audits were events — auditors would arrive quarterly or annually, collect evidence at that point in time, evaluate, and leave. Two problems: First, no one knows the actual state between audits (point-in-time illusion). Second, evidence collection is manual drudgery, expensive and error-prone — taking screenshots, capturing policies, exporting logs, pasting into spreadsheets.

Continuous compliance transforms this into state — evidence is collected continuously and automatically, so "are we SOC2-compliant right now" can be answered with data at any moment. This is the compliance version of DevOps transforming "release as event" into "continuous delivery flow." Audit Manager is the tool enabling this transition.

💡 Related Theory: Continuous compliance is the cloud-native evolution of GRC (Governance, Risk, Compliance). Traditional GRC tools (Archer, ServiceNow GRC) required humans to input and manage controls, evidence, and risks. In the cloud, evidence is API-exposed (Config, CloudTrail, Security Hub already know state), so evidence collection can be automated. The core concept is mapping controls to evidence — regulatory frameworks (SOC2's trust service criteria, HIPAA's security rules) define abstract controls ("access must follow least privilege"), and concrete evidence proving compliance (IAM policy snapshots, access logs) is automatically collected and mapped. This mapping bridges "abstract regulation → concrete technical evidence," and Audit Manager's Framework pre-lays this bridge.

AWS Audit Manager — Auto-Mapping Evidence to Controls

Audit Manager provides pre-defined Frameworks (SOC2, HIPAA, PCI DSS, GDPR, ISO 27001, etc.). A Framework contains the regulatory controls, and for each control, evidence is automatically collected — pulling evidence from CloudTrail (API activity), Config (resource state), and Security Hub (security findings) and mapping it to the control. Custom Frameworks can also be created, and ultimately an Assessment Report is generated for auditors.

aws auditmanager create-assessment --name SOC2-2026 \
  --framework-id <framework-id> \
  --assessment-reports-destination destinationType=S3,destination=s3://audit-reports/ \
  --roles roleType=PROCESS_OWNER,roleArn=arn:aws:iam::...:role/AuditOwner \
  --scope '{"awsAccounts":[{"id":"PROD-ACCT"}]}'

⚠️ Pitfall: Don't confuse Audit Manager with Security Hub/Config roles. Config/Security Hub evaluate "are we compliant right now" and remediate violations — they're operational tools. Audit Manager collects and organizes evidence to prove compliance to auditors — it's an audit preparation tool. If Config says "S3 is encrypted," Audit Manager says "S3 is encrypted — here's the evidence (Config snapshot, CloudTrail logs) attached to the relevant SOC2 control, packaged in a report." In exams: "auto-fix violations" is Config; "auto-collect audit evidence and create reports" is Audit Manager.

Amazon Macie — Pattern Matching and ML for Data Classification

Macie automatically discovers sensitive data in S3 objects. It classifies data using pre-defined identifiers (credit card numbers, U.S. SSNs, passport numbers, medical records, etc.) and custom identifiers (regex), emitting Findings to Security Hub when discovered.

The core mechanism is two-layered. Deterministic pattern matching (regex + validation algorithms) catches structured identifiers like card numbers and SSNs, while machine learning and contextual analysis makes non-structural judgments like "does this text block look like a medical record?"

🔍 Deeper: Credit card detection shows the sophistication of data classification. Card numbers must satisfy the Luhn algorithm (1954, Hans Peter Luhn at IBM, ISO/IEC 7812 standardized) — a checksum where the last digit (check digit) must match a value computed from preceding digits. Macie finds 16-digit patterns, then runs Luhn validation to filter false positives (accidentally 16-digit numbers like order IDs). SSNs also validate against issuance rules. This "pattern + validation" dual structure determines classification accuracy — pattern alone causes false positive explosions. This exemplifies the classic text-processing tradeoff between regex and structured validation.

💡 Related Theory: Macie solves the problem in data governance of "know your data." Regulations like GDPR (2018), CCPA (California), HIPAA demand "you must know where PII is, protect it, and delete it if requested (right to be forgotten)." But finding PII in petabyte-scale lakes is impossible manually — this is the dark data problem. Macie illuminates this darkness through automated discovery. The key pattern is "classify before you protect" — without knowing what's sensitive, you can't protect appropriately. Macie's Finding goes to Security Hub and surfaces "SSN in public bucket" as Critical — this is the chain from classification → risk assessment → response. Lesson: response speed is proportional to knowing what you use; asset and dependency inventories are security's prerequisite.

aws macie2 enable-macie
aws macie2 create-classification-job --job-type ONE_TIME \
  --name pii-scan-prod \
  --s3-job-definition '{"bucketDefinitions":[{"accountId":"111","buckets":["prod-data"]}]}'

⚠️ Pitfall: Macie charges per-object, per-storage-unit, and large data lakes can be costly. Running full-scan Discovery Jobs every time balloons costs. Industry practice separates ①automated sensitive data discovery (sampling-based continuous monitoring, low cost) and ②full Discovery Jobs (precise, high cost), scanning only new/changed objects incrementally to control costs. In exams, "large lake PII scan" pitfall is usually "full scan repeats without cost consideration."

Inspector v2 — CVE and Vulnerability Management Standardization

Inspector v2 (re-released 2021) scans EC2, ECR container images, and Lambda across the board to find CVEs (Common Vulnerabilities and Exposures).

The key is continuous monitoring — not scan-once-and-done, but auto-re-evaluating already-scanned assets when new CVEs are published. "Yesterday's image was safe, but today's published CVE makes it vulnerable" is caught automatically. Findings integrate to Security Hub.

🔍 Deeper: Understanding CVE and CVSS history reveals vulnerability management's structure. CVE, started by MITRE in 1999, is a "dictionary assigning unique identifiers to known vulnerabilities (like CVE-2021-44228)." Before CVEs, vendors called the same vulnerability different names — chaos. CVE created "a common language for vulnerabilities." CVSS (see Day 2) scores severity 0-10. NVD (National Vulnerability Database, NIST-run) is the U.S. government DB attaching CVSS scores and impact analysis to CVEs. Inspector consumes this ecosystem — it extracts asset package inventory, cross-references CVE feeds, scores severity from NVD's CVSS, and reports via ASFF to Security Hub. Inspector's value lies not in "scanning engine" but in "continuously-updated CVE feed × my asset inventory cross at all times."

📚 Case Study: Log4Shell (CVE-2021-44228, CVSS 10.0), disclosed December 2021, is a remote code execution in Java's Log4j logging library. Nearly every Java app depended on Log4j directly or indirectly, making it "one of the broadest vulnerabilities in internet history." The core lesson: dependency visibility. Many organizations couldn't instantly answer "which services use Log4j?" — late response. This is why SBOM (Software Bill of Materials, software component inventory) and SLSA (Supply-chain Levels for Software Artifacts, Google 2021, "salsa") became critical. SBOM is "list of all components in this build," answering "where does the vulnerable component live?" instantly during Log4Shell-style incidents. Lesson: response speed correlates with knowing what you use — asset and dependency inventory is security's prerequisite.

aws inspector2 enable --resource-types EC2 ECR LAMBDA

🔍 Deeper: Post-Log4Shell, SLSA (Google 2021) is a software supply-chain integrity framework. It proves build provenance, ensuring "this artifact really built from trusted source without tampering" (levels 1-4). ECR already combines image signing (AWS Signer), immutable tags, and Inspector enhanced scanning to harden container supply chains. In DOP context, "ECR push with automatic CVE scan (Inspector enhanced scanning) + signature validation + immutable tags" is the container supply-chain security standard pattern.

IAM Access Analyzer — Proving Least Privilege with Data

IAM Access Analyzer does four things.

🔍 Deeper: External Access Findings' internal engine is interesting. Access Analyzer uses automated reasoning — using mathematical logic to "prove" whether a policy permits external access. AWS's Zelkova engine converts IAM policies into logical formulas that SMT (Satisfiability Modulo Theories) solvers can solve, mathematically proving "does a case exist where external principals can access under this policy?" This is formal verification, not empirical "test the policy" — it reasons about all possible requests and guarantees correctness. No "unlucky missed case" exists — if the policy allows external access, it's found. Automated reasoning is AWS's core tech used in multiple places (S3 Block Public Access, VPC Reachability Analyzer, etc.) in AWS's "Provable Security" initiative.

💡 Related Theory: Policy Generation and Unused Access implement the Principle of Least Privilege with data. Least Privilege, established by Saltzer & Schroeder's 1975 seminal paper "The Protection of Information in Computer Systems" as one of 8 core security design principles, says "subjects should possess only privileges strictly necessary for their task." The practical problem: "strictly necessary" is hard for humans to determine, so permissions usually end up oversized (privilege creep) and never shrink. Access Analyzer inverts this. Policy Generation extracts only actually-used permissions from CloudTrail, creating policies (observation-based least privilege); Unused Access finds unused permissions for removal. "Achieve least privilege through observation, not guessing" is the principle.

# Organization-wide analyzer
aws accessanalyzer create-analyzer --analyzer-name org-analyzer --type ORGANIZATION
 
# CloudTrail-based policy generation (actual-use permissions only)
aws accessanalyzer start-policy-generation \
  --policy-generation-details principalArn=arn:aws:iam::...:role/Analyst,cloudTrailDetails={accessRole=...,trails=[...],startTime=...}

Firewall Manager and CloudTrail Lake — Completing the Stack

Firewall Manager manages multi-account WAF, Shield, Network Firewall, and security group policies centrally. Integrated with Organizations, it auto-applies policies to new accounts and resources — "enforce this WAF rule on every ALB across the org" becomes one-time setup.

CloudTrail Lake is a SQL-queryable data store of CloudTrail events, retaining up to 7 years and unifying multi-account and multi-region. Security incident investigation becomes answering "what API did this credential call 3 years ago" with SQL.

💡 Related Theory: Mapping this entire stack to NIST's Cybersecurity Framework (CSF) 5 functions — Identify, Protect, Detect, Respond, Recover — completes the picture. Identify: Config (inventory), Macie (data classification), Access Analyzer (exposure/unused). Protect: Firewall Manager (WAF/SG), Conformance Pack. Detect: GuardDuty (threats), Inspector (vulnerabilities), Security Hub (integration). Respond: EventBridge→SSM/Lambda auto-remediation. Recover: Snapshots, backups. Knowing which AWS service fills which CSF function lets you quickly diagnose in scenarios "what's missing and what should fill it."

Comprehensive Security Automation Stack (NIST CSF Mapping)
==================================================
  Identify   ├─ Config(inventory) ├─ Macie(data) └─ Access Analyzer(exposure·unused)
  Protect    ├─ Firewall Manager ├─ Conformance Pack └─ KMS/encryption
  Detect     ├─ GuardDuty(threats) ├─ Inspector(vulnerabilities) └─ Security Hub(integration·ASFF)
  Respond    └─ EventBridge → SSM Automation / Lambda
  Recover    └─ Snapshots / Backups / PITR
  Forensics  └─ CloudTrail Lake (7y, SQL)

🎯 Scenario: "Fintech org needs ①continuous SOC2/PCI audit evidence ②auto-discover card numbers and SSNs in data lake S3, cost-controlled ③ECR push auto-scans npm/pip dependency CVEs + auto-re-evaluates on new CVEs ④discover S3/roles exposed to external accounts via formal proof ⑤shrink over-granted IAM permissions to observation-based least privilege ⑥7-year security incident SQL investigation." → ① Audit Manager Frameworks (SOC2, PCI) + Assessment Report ② Macie auto-discovery (sampling continuous) + incremental Discovery Job for new/changed (cost-controlled) ③ Inspector v2 ECR enhanced scanning (continuous monitoring) ④ Access Analyzer External Findings (Zelkova formal reasoning) ⑤ Access Analyzer Policy Generation (CloudTrail-based) + Unused Access ⑥ CloudTrail Lake (7-year SQL). All Findings integrated ASFF→Security Hub, violations routed EventBridge→SSM auto-remediation.

Summary

Today we saw four key points. First, continuous compliance transforms audits from event to state — a GRC cloud evolution. Audit Manager auto-maps evidence (Config, CloudTrail, Security Hub) to regulatory controls and generates Assessment Reports, proving compliance. Unlike Config (operational "are we compliant now"), Audit Manager is audit preparation ("evidence for auditors"). Second, Macie classifies dark data's PII/PHI with pattern matching (Luhn validation, etc.) + ML, implementing "classify before protect" data governance, though per-object cost control is a pitfall. Third, Inspector v2 consumes the CVE/CVSS/NVD ecosystem for continuous scanning (EC2, container, Lambda), auto-re-evaluating on new CVEs, and Log4Shell's lesson surfaces SBOM and SLSA supply-chain security. Fourth, Access Analyzer proves external exposure through Zelkova formal reasoning and observation-based least privilege via Policy Generation, completing NIST CSF 5 functions with Firewall Manager (central policy) and CloudTrail Lake (7-year SQL).

Next we'll synthesize Week 14 overall with scenario questions verifying end-to-end application.

📌 Today's Summary

Today's key points were four-fold. First, continuous compliance transforms audits from event to state — a GRC cloud-native evolution. Audit Manager auto-maps evidence (Config, CloudTrail, Security Hub) to regulatory Framework controls and generates Assessment Reports proving compliance ("evidence collection for auditors" is its role). Second, Macie classifies dark data's PII/PHI with pattern matching (Luhn, etc.) + ML implementing "classify before protect," though per-object billing is a cost pitfall. Third, Inspector v2 consumes CVE/CVSS/NVD ecosystem for continuous scanning (EC2, container, Lambda), auto-re-evaluating on new CVEs, and Log4Shell's lesson surfaces SBOM and SLSA supply-chain security importance. Fourth, Access Analyzer proves external exposure via Zelkova formal reasoning and observation-based least privilege via Policy Generation, completing NIST CSF 5 functions with Firewall Manager (central policy) and CloudTrail Lake (7-year SQL investigation).