Day 5 - Week 12 Integrated Review: Wiring the Whole Nervous System of Incident Automation Together

GuardDuty detected a Critical threat (a compromised EC2 instance communicating with a C2 server). You want to build a flow that automatically isolates the instance and takes a forensic snapshot, then runs termination only after operator approval. Which is the most suitable?

• AA single Lambda runs detection, isolation, snapshot, and termination all immediately and automatically, minimizing MTTR with no human intervention
• BEventBridge pattern-matches the GuardDuty Finding (by severity) → SSM Automation Runbook (isolation and snapshot with aws:executeAwsApi, a termination gate with aws:approve)
• CA Lambda polls CloudTrail logs every minute, finds traces of C2 communication, and runs isolation, snapshot, and termination
• DSend the GuardDuty alert only as an SNS email and have operators handle isolation, snapshot, and termination entirely by hand

You want to build a flow that polls messages from an SQS queue, does first-pass filtering with an event pattern, enriches the missing information through an external API, and stores the result in DynamoDB, all without glue code. Which is the most suitable?

• AWrite a Lambda yourself that handles SQS long polling, event filtering, external API enrichment, and DynamoDB storage in a single function
• BEventBridge Pipes (Source: SQS → Filter → Enrichment → Target: DynamoDB)
• CSpecify SQS directly as the event source of an EventBridge Rule, enrich with an Input Transformer, and use DynamoDB as the target
• DDistribute messages to multiple subscribers with SNS fan-out, and have each one filter, enrich, and store to DynamoDB

An auto-recovery Lambda isolates the same instance twice because of duplicate delivery of the same alarm, is triggered repeatedly at short intervals and exhausts resources, and touches too many instances at once. Which combination of safety nets stops the runaway?

• AIncrease the Lambda timeout and memory so the isolation action is guaranteed to finish without being cut off midway
• BIdempotent (DynamoDB ConditionExpression) + Cooldown (lastActionTime, no reprocessing within T seconds) + Bounded Action (a cap of at most N per trigger)
• CSet the EventBridge RetryPolicy to 0 to block duplicate triggers caused by retries at the source
• DLengthen the CloudWatch Alarm evaluation period to lower the frequency of duplicate and consecutive triggers itself

Every day at exactly 3:00 AM, 50,000 per-user backup jobs run simultaneously and the downstream collapses under the load. Also, there are hundreds of thousands of such per-user schedules, so you hit the CloudWatch cron rule count limit. What is the correct solution?

• AMove the cron time up to 02:59 to shift to a quieter period for the downstream and avoid the load collision
• BEventBridge Scheduler: supports millions of schedules + Flexible Time Window spreads executions across the window (injecting jitter, preventing the thundering herd)
• CGreatly increase the Lambda reserved concurrency limit so that all 50,000 concurrent executions are accommodated
• DManually divide the jobs into 10 groups and place them at different cron times to spread the load over time

In a Critical incident, if the primary on-call person does not respond within 30 minutes, you want to automatically page the secondary group, and if there is still no response, the manager, with the whole process recorded automatically on a timeline. What is the AWS-native solution?

• ASubscribe all on-call staff to an SNS topic to notify them at the same time and let whoever responds first handle it
• BAWS Incident Manager: Escalation Plan (next stage after N minutes without a response) + Engagement (SMS/Voice/Email) + Timeline (automatic recording) + PIR
• CA Lambda polls every 30 minutes for whether anyone responded and, if not, sends SMS sequentially to the next stage's group
• DCreate several CloudWatch Alarms per stage and notify the primary, secondary, and manager with time offsets

In Slack, you want operators to freely run diagnostic commands through Chatbot, but never be able to terminate prod EC2 instances or delete RDS even by mistake. You want to keep the IAM Role broad for operational convenience. What is the correct configuration?

• ARestrict the IAM Role used by Chatbot to read-only permissions so it cannot make any change or deletion
• BAn explicit Deny (ec2:TerminateInstances, rds:Delete*) in the Chatbot channel Guardrail Policy: blocked regardless of the IAM Role
• CSwitch the Slack channel to private and invite-only so that only privileged operators can type dangerous commands
• DTurn off Chatbot's command execution (run command) feature and use it only for lookups and receiving notifications

An EC2 instance is unresponsive because of a failure in the underlying host hardware. You want to recover it automatically onto healthy hardware while keeping the same IP, ENI, and EBS. What is the correct configuration?

• APut it in an ASG and set health-check-type to EC2
• BStatusCheckFailed_System metric alarm + ec2:recover alarm action
• CStatusCheckFailed_Instance alarm + terminate and recreate with Lambda
• DFail over with a Route 53 health check

You want to build a recovery in which branching, waiting, retries, and compensation are intertwined: "RDS failover → wait for completion → cache invalidation → health check → on failure escalate to a human, on success report to Slack + create a ticket." Which is the most suitable?

• AImplement failover, invalidation, health check, and escalation all in a single Lambda with if-else branches and sleep waits
• BA Step Functions State Machine: Retry/Catch on each step, state preservation, Saga compensation
• CMake each step an EventBridge Rule and chain them so that one step's completion event triggers the next Rule
• DA Lambda polling every minute by cron checks the current recovery stage and decides the next action to proceed

You want to announce CodePipeline failures to a Slack channel in the simplest, most standard way. Which is the most suitable?

• ATrigger a Lambda on every pipeline failure event to call the Slack Incoming Webhook directly
• BCodeStar Notifications rule → AWS Chatbot (Slack channel) target
• CA cron Lambda that polls CodePipeline's CloudWatch Logs finds failure logs and posts them to Slack
• DAdd a manual approval stage to the pipeline so that a Slack notification goes to the approver on failure

The operations team built auto-recovery logic, but when a real failure first occurred, it did not work because of a bug (like the restart path in the 2017 S3 incident). What is the standard approach for discovering such "rusty automation" in normal times?

• AWrite more automation code to cover every possible failure case in advance without gaps
• BRegularly inject failures with AWS FIS and rehearse auto-recovery, notifications, and runbooks as a whole in a Game Day (chaos engineering)
• CTurn off the unreliable auto-recovery in production and run only the proven human-centered manual response
• DLengthen the retention period of the related logs so that automation failures can be traced after the fact

Payment events processed through EventBridge are occasionally processed twice, causing double billing. What are the root cause and the solution?

• AEventBridge delivering events in duplicate is a bug, so contact AWS Support and request a fix
• BEventBridge delivery is at-least-once, so duplicates are normal: add idempotency logic to the consumer (duplicate blocking based on the event ID: DynamoDB conditional write, Lambda Powertools Idempotency)
• CSet the EventBridge target's RetryPolicy to 0 to eliminate duplicate delivery caused by retries at the source
• DRemove the payment consumer's DLQ to simplify the path so that failed events are not re-ingested and reprocessed

You want to send incidents securely, with authentication, from EventBridge to an external SaaS (PagerDuty), without hand-coding credential refresh, rate limiting, retries, or secret management. Which is the most suitable?

• AA Lambda calls the PagerDuty REST API directly and handles authentication by storing the API Key in an environment variable
• BEventBridge API Destination + Connection (authentication managed through Secrets Manager, rate limiting, using EventBridge retries and DLQ)
• CSend the incident notification through an SNS Email subscription to PagerDuty's email integration address
• DWrite the events to S3 and have PagerDuty poll the bucket periodically to create incidents