Day 2 - Integrated Review of Domains 3 & 4: Operations & Support + Security & Governance

📌 Key Takeaways

• Domain 3 (operations and support, 22%) and Domain 4 (security and governance, 18%) ask "how do you detect and recover from failures" and "who accesses the data and how is it encrypted."
• For orchestration: an existing Airflow DAG = MWAA, coordinating heterogeneous AWS services = Step Functions, Glue-only = Glue Workflows.
• Observability has three branches: threshold alerts = CloudWatch Alarm, root cause tracing = CloudWatch Logs Insights, who made the call = CloudTrail. Data quality is verified, and bad data isolated, with Glue Data Quality (DQDL) rules.
• Permissions live in different layers: services and APIs get least-privilege IAM roles, tables, columns, and rows are handled by Lake Formation, and encryption keys are governed by KMS key policies. The data is readable only if all three gates are passed.
• When you need key control and auditing, use SSE-KMS (customer managed key); automatic detection of sensitive data is Macie; hiding values at query time is Lake Formation filters and Redshift dynamic data masking.

Viewing Operations and Security as One Bundle

Yesterday we reviewed the pipeline through which data flows. Today we bundle the areas that run that pipeline reliably and protect it safely. The two domains connect on a single pipeline like this.

1. Operations: MWAA/Step Functions coordinate the pipeline → CloudWatch monitors metrics and logs → alarms and retries on failure
2. Quality: validate before loading with Glue Data Quality rules, isolate violating data
3. Security: least-privilege IAM roles → encrypt stored data with KMS, encrypt the in-transit segment with TLS
4. Governance: detect sensitive information with Macie → fine-grained access with Lake Formation → audit with CloudTrail

Expanded into the exam guide's tasks, it looks like this.

Domain 3: Orchestration

Orchestration is tying multiple jobs together according to order and dependencies.

💡 Related theory: If you "want to keep using an existing Airflow DAG as is," MWAA is the answer; to "coordinate heterogeneous AWS services (Lambda, ECS, Glue) with branching, retries, and parallelism," Step Functions is the answer. If you are only chaining Glue jobs, Glue Workflows is enough.

This is another point where the choice splits from an operations standpoint (Week 5 Day 5).

• Idle cost: MWAA is billed while the environment is up. Step Functions Standard is billed by the number of state transitions, and Express by the number of executions, duration, and memory. For "a short flow that runs once a day, with no idle cost," it is Step Functions.
• Waiting for completion: The integration pattern that waits until a Glue job or EMR step finishes (.sync) works only in Standard. Express runs for at most 5 minutes, so it is not the default for ETL orchestration.
• Where retries live: If you stack Step Functions Retry and the Glue job's own retry, they multiply. Put retries in one layer only, and branch to notification (SNS) or compensation handling with Catch.
• Diagnosing MWAA failures: If a DAG does not appear in the list or a task does not start, check in CloudWatch Logs the DAG path in S3, the permissions of the environment execution role, and errors from installing required packages.

Domain 3: Monitoring and Logging

• CloudWatch Metrics/Alarms: Threshold alarms on job failure rate, latency, and resource usage.
• CloudWatch Logs: Aggregation of Glue, Lambda, and EMR execution logs and Logs Insights queries.
• CloudTrail: API call audit log (who did what and when). Also connected to security and governance.
• EventBridge: Notifications and follow-up triggers from job state change events.

The three kinds of pipeline observability:
- Metric  → "notify when a threshold is exceeded" with a CloudWatch Alarm
- Log     → "trace the cause" with CloudWatch Logs Insights
- Audit   → "who performed this action" with CloudTrail

💡 Related theory: Separate them this way: "trace why it failed" is CloudWatch Logs, "threshold-based alert" is a CloudWatch Alarm, and "trace who called this API" is CloudTrail. Not mixing these three up is a scoring point in Domain 3.

Even when everyone says "leave a record," each tool records something different.

• "Who read the S3 object" is a data event, not a management event. If you have not turned on data events for that bucket in the trail, there is no record. They are off by default because the volume adds cost.
• If you need audit records older than 90 days, use a trail (store in S3 and analyze with Athena) or CloudTrail Lake. If the cue is "query audit logs centrally with SQL," it is CloudTrail Lake.
• When log volume is very large, the analysis tool changes. Ad hoc exploration is Logs Insights, large volumes of logs piled up in S3 are Athena, search and dashboards are OpenSearch Service, and transforming and aggregating very large logs is EMR.

Domain 3: Data Quality

• Glue Data Quality: Define rules (DQDL) on a dataset to verify completeness, validity, and duplicates. When a rule is violated, stop the job or isolate the data.
• DataBrew: Use profiling to visually grasp missing values, outliers, and distributions.
• Idempotency and retries: Design jobs to be idempotent so reprocessing after a failure is safe.

# Glue Data Quality rule (DQDL) example
ruleset = """
Rules = [
   IsComplete "order_id",
   ColumnValues "amount" >= 0,
   Uniqueness "order_id" > 0.99
]
"""
# If the rules pass, load; if they fail, route to the quarantine bucket

The quality task contains two more concepts besides defining rules.

• Whether the schema is right (shape) and whether the values are right (content) are different problems. Negative amounts, duplicate keys, and NULLs are caught by Data Quality, not by Schema Registry (Week 4 Day 4).

Domain 3: The Scale Between Provisioned and Serverless

The analysis task asks not about service names but "when do you keep always-on capacity and when do you pay for what you use."

• Athena cost control is done in a workgroup. A query that exceeds the per-query scan limit is canceled. A workgroup-wide usage limit only notifies via SNS when the threshold is crossed; it does not automatically cancel running queries.
• Visualization is QuickSight. If a dashboard queries the source every time and is slow or expensive, importing the data into the in-memory engine (SPICE) becomes the answer.

Domain 4: Authentication and Authorization (IAM)

• IAM roles: Grant services (Glue/EMR/Lambda) least privilege. Do not keep human credentials in code.
• Least privilege principle: Allow only the necessary actions and resources. No overuse of *.
• Resource-based policies: S3 bucket policies, KMS key policies, and so on allow access from the resource side.
• Relationship between Lake Formation permissions and IAM: Access to data lake data is handled by LF, and service operation permissions by IAM.

If "a Glue job must access S3 and KMS," grant those permissions to the job's execution role (IAM Role). A choice that hardcodes access keys in the job script is wrong.

• In policy evaluation, an explicit deny takes precedence over any allow. For a principal in a different account to get access, both sides (the requester's IAM policy and the resource-side policy) must allow it.
• Credentials such as database passwords are kept in a store, not in code.

• Permissions inside Redshift are granted with GRANT to the database's users, groups, and roles, not through IAM. To let data be read across accounts or clusters without copying, use Redshift data sharing.
• The network layer is also part of authentication. If you cannot connect to a data source, look at the security group's inbound rules; if you must reach an AWS service without going through the internet, look at a VPC endpoint (PrivateLink).

Domain 4: Encryption (KMS)

• Encryption at rest: S3 (SSE-S3/SSE-KMS), Redshift, RDS, EBS, and Kinesis all support server-side encryption, and when you need key control, you specify a KMS key.
• Encryption in transit: TLS/HTTPS.
• SSE-KMS vs SSE-S3: Use SSE-KMS when you need to manage, audit, and rotate keys yourself and need per-key access control; use SSE-S3 for simple managed encryption.
• Customer managed key (CMK): Fine-grained control over who can encrypt and decrypt through the key policy, and auditing of key use through CloudTrail.

What you can do with a KMS key differs by who manages it.

• The rotation period of AWS managed keys changed from 3 years to 1 year in May 2022. Exam material may still say 3 years.
• Even when a key is rotated, the key ID and ARN stay the same. The old key material is kept, so past ciphertext is decrypted as before, and existing data is not re-encrypted. There is no need to change application code.
• Automatic rotation works only for symmetric encryption keys that use key material created by KMS. For asymmetric keys, HMAC keys, and keys in custom key stores, you rotate manually by creating a new key and moving the alias.
• Since January 5, 2023, S3 encrypts all new objects with SSE-S3 by default. Read the question not as "fix an unencrypted bucket" but as "who controls the key."
• When objects are read and written in bulk with SSE-KMS, KMS request cost and the per-second request limit become a problem. Turning on an S3 Bucket Key creates data keys from a bucket-level key and greatly reduces KMS requests (it cannot be used with DSSE-KMS).
• A trap in cross-account sharing: even if you open the bucket policy, if the object is encrypted with the AWS managed key (aws/s3), the other account cannot decrypt it. You must encrypt with a customer managed key and add the other account to the key policy.
• The in-transit segment is enforced through settings. S3 rejects requests where aws:SecureTransport is false in the bucket policy, and Redshift accepts only TLS connections through require_ssl in the parameter group.

Domain 4: Sensitive Data Detection and Data Protection

• Amazon Macie: Uses machine learning and pattern matching to automatically discover and classify sensitive data such as PII (personally identifiable information) and credit card numbers in S3. It solves "we don't know where the sensitive data is."
• Masking/tokenization: Hide sensitive columns at output with Glue, Athena, and Redshift Dynamic Data Masking.
• Data classification: A protection chain running discovery (Macie) → classification → permissions (Lake Formation) → encryption (KMS).

For the scenario "sensitive information (PII) is scattered all over our S3 buckets and we want to find and classify it automatically," the answer is Amazon Macie. Macie is "detection/classification"; access control is Lake Formation/IAM, and protection is handled by KMS and masking, a division of roles.

When several choices say "handles sensitive information," separate them by where and when they act.

• Macie detects in two ways. Automated sensitive data discovery samples representative objects from buckets and scans broadly, while a discovery job analyzes specified buckets deeply. For in-house formats that the built-in (managed) identifiers do not catch, use regex-based custom identifiers.
• Macie sends findings to EventBridge. "If PII is found, quarantine and notify automatically" is the combination Macie → EventBridge → Lambda/SNS.
• A data sovereignty requirement ("backups and replicas must not go to a disallowed Region") is a matter of blocking, not detection. Apply a policy that denies operations in that Region using a Region condition (aws:RequestedRegion).

The Three Gates a Single Query Must Pass

A confusing type in security questions is "I granted the permission but still get AccessDenied." A single query that reads a Lake Formation managed table with Athena passes through three layers in turn.

① IAM          permission to run the query and read the catalog + lakeformation:GetDataAccess
      ↓
② Lake Formation   has SELECT on the table, columns, and rows been granted to that principal?
      ↓            (if it passes, it issues temporary credentials for the registered S3 location)
③ KMS          if the object is SSE-KMS, can it be decrypted with that key (key policy)?

• For data in a location registered with Lake Formation, you do not give the user role S3 read permissions directly. Instead, lakeformation:GetDataAccess must be in IAM. The bucket where query results are written is separate, so S3 permissions on that side are still required.
• Unregistered locations are controlled the old way, by IAM and bucket policies alone. Hybrid access mode enforces Lake Formation permissions only on principals that have opted in for the same table, while the rest keep accessing through IAM permissions, which lets you migrate gradually without breaking existing workloads.
• Which gate blocked you can be sorted out from the deny events in CloudTrail (the error code and the calling API).

Symptom → Cause → Action

Scenario → Answer Decision Table

Tomorrow we match the real exam pace with comprehensive scenarios that mix all four domains.

📖 Glossary

• Management events / data events: Calls that create and change resources, versus calls that read and write the data inside resources. CloudTrail records only the former by default.
• Metric filter: A setting that turns the number of times a particular pattern appears in a log into a CloudWatch metric. It is the path for putting an alarm on logs.
• Data skew: A phenomenon where data piles up on particular keys so that only some tasks slow down or fail.
• Salting: A technique that appends a random value to a skewed key to spread it across several partitions.
• Key policy: The resource-based policy attached to a KMS key. It decides who can encrypt and decrypt with that key.
• Envelope encryption: A method where data is encrypted with a data key and the data key is encrypted with a KMS key. This is how SSE-KMS works.
• S3 Bucket Key: A bucket-level key used briefly in SSE-KMS so that KMS is not called for every object. It reduces KMS request cost.
• Data filter: In Lake Formation, a definition of which rows and columns to show a principal. It is the unit of row-, column-, and cell-level security.
• Dynamic data masking: A method that leaves the stored value as is and shows masked results depending on who is querying.
• Hybrid access mode: A Lake Formation setting that keeps both the Lake Formation permission path and the IAM permission path on the same catalog resource.