Day 5 - Week 8 Comprehensive Review: Data Operations and Support Recap

📌 Key Takeaways

• Operations problems split into four questions: Is it healthy right now? (CloudWatch) / Is the data right? (Glue Data Quality) / Why, and who broke it? (logs and CloudTrail) / How much does it cost? (cost tools).
• "Who did it" is CloudTrail; "how the system behaved" is CloudWatch Logs and Metrics. The key streaming-lag metric is IteratorAgeMilliseconds.
• The quality rule language is DQDL, and for data whose volume varies you add anomaly detection. Block and quarantine anything below the standard with a Step Functions Choice state.
• Retry only idempotent work, with exponential backoff and jitter. Move repeated failures to a DLQ and redrive after fixing the cause.
• Look at cost with allocation tags, handle variation with auto scaling, and handle the baseline with sizing and commitments. Sizing comes first.

The Operations Map Through Four Questions

This week covered the stage where a pipeline, once "built," is "operated." The four days' topics are not separate; they are the four questions an operator asks in turn while going through a single incident.

                     Pipeline in operation
                            │
   ┌──────────────┬─────────┴────────┬───────────────┐
 Is it healthy?   Is the data right?  Why/who broke it?  What does it cost?
   (Day 1)          (Day 2)            (Day 3)           (Day 4)
   │                │                  │                 │
 CloudWatch       Glue Data          CloudWatch Logs   Cost allocation tags
 metrics, alarms  Quality            CloudTrail        Cost Explorer
 EMF, composite   DQDL, anomaly      Retry/Catch, DLQ  Budgets, anomaly
 alarms           detection                            detection

• The first two questions on the left are detection, the third is diagnosis and recovery, and the fourth is sustainability. When reading a problem, first decide which of the four the question is asking, and you can narrow the choices.
• A common wrong-answer structure is "a tool from the wrong column." It makes you pick CloudWatch Logs for "who deleted it" and CloudTrail for "detect a cost spike."

Signal → Tool Mapping

• "The job succeeded" and "the data is right" are different facts. A job can read an empty file and still succeed. That is why monitoring (Day 1) cannot substitute for quality validation (Day 2).

Distinguishing Look-Alikes

Records and logs

Three features that catch "different from usual"

• All three solve the same problem that "it is hard for a person to set a threshold." When all three appear together in the choices, pick by what is anomalous.

Four mechanisms for handling failure

• The first three are mechanisms for when the execution fails; the fourth is for when the execution succeeded but the result is wrong.

Features that adjust capacity automatically

Week 8 Numbers and Defaults on One Page

• Two values have changed recently: the default Glue job timeout (48 hours → 8 hours from 5.0) and whether cost allocation tags apply retroactively (not possible → backfill possible). Exam material may present the old values.

Integrated Scenario: Operating a Nightly Order ETL

00:00 EventBridge schedule → Step Functions pipeline starts
  1) Glue ingestion job (real-time logs + Auto Scaling + job metrics)
  2) Glue Data Quality evaluation (DQDL + anomaly detection)
       └─ Critical rule fails or score < 0.95 → quarantine bucket + SNS + stop
  3) EMR transform (core On-Demand, task Spot, Managed Scaling)
  4) Load to curated + Athena validation query
 
Cost operations:
  - Project/Environment tags on every resource, activated in the management account
  - Budgets 80% alert, Cost Anomaly Detection for spike detection
  - Monthly Cost Explorer review to readjust sizing

Following One Incident Through the Four Questions

Suppose something went wrong on the same pipeline before dawn. The four days' content connects in chronological order.

• The alarm fired at 02:10 because missing-data handling had been set on the processed count metric. With the default, a person would have found it in the morning.
• Curated was not contaminated at 02:15 because the gate was before the load. Without the gate, what happened after 02:30 would have been "investigating the extent of contamination."
• The reprocessing at 03:00 did not create duplicates because the load was a partition overwrite.

Common Pitfalls

• A quiet alarm is not evidence of health. If the metric stops, an alarm with the default setting does not fire.
• The quality score is not a row ratio. It is the ratio of rules that passed, so judge critical rules separately from the score.
• Anomaly detection does not work from day one. Statistics must accumulate, and an anomalous value caught once is learned as normal unless you exclude it.
• Even with CloudTrail on, object access is not recorded. Data events are a separate setting.
• Jitter in Step Functions is off by default. Backoff alone does not scatter concurrent retries.
• The DLQ of a standard queue does not restart the retention period. Set DLQ retention longer than the source queue's.
• A tag does not show up in cost unless activated, even if attached. In an organization, that is the management account's job.
• An Athena workgroup alert does not stop queries. Only the per-query limit does.
• Raising the scaling upper bound does not reduce the usual cost. Redo the baseline.

Question Clue → Answer Lookup Table

Next week is data security and governance. We start with access control in IAM and Lake Formation.

📖 Glossary

• Observability: The degree to which you can tell from the outside what is happening inside a system through metrics, logs, and traces.
• Symptom-based alarm: An alarm placed not on a resource number such as CPU but on a phenomenon users experience, such as lag or failure rate.
• Validation gate: A point that does not pass data whose quality falls short of the standard to the next stage.
• Quarantine bucket: A place that collects data that failed the gate, together with the failure reasons.
• Idempotent reprocessing: Reprocessing where running the same interval several times gives the same result as running it once.
• redrive: Putting failed messages or executions back on the processing path after fixing the cause.
• Baseline: The base capacity fitted to the normal load. Distinguish it from the variation that scaling absorbs.
• Runbook: A procedure document that records what to check and what action to take, in what order, when a particular failure occurs.