Cert Notes/ Commute Study Notes
Roadmap
KOEN
CLF-C02 · FoundationalCloud Practitioner - Foundational
DVA-C02 · AssociateDeveloper - Associate
SAA-C03 · AssociateSolutions Architect - Associate
SOA-C02 · AssociateCloudOps Engineer - Associate
SAP-C02 · ProfessionalSolutions Architect - Professional
DOP-C02 · ProfessionalDevOps Engineer - Professional
SCS-C03 · SpecialtySecurity - Specialty
MLA-C01 · AssociateMachine Learning Engineer - Associate
AIF-C01 · FoundationalAI Practitioner - Foundational
DEA-C01 · AssociateData Engineer - Associate
  • Week 1
    • 1.What Is Data Engineering
    • 2.Batch vs Streaming
    • 3.A Bird's-Eye View of AWS Data Services
    • 4.Data Formats and Modeling
    • 5.Week 1 Comprehensive Review
  • Week 2
    • 1.Batch Ingestion: S3 Upload, DataSync, Transfer Family, Snow
    • 2.Kinesis Data Streams: Shards, Partition Keys, and Throughput
    • 3.Kinesis Data Firehose: Delivery Streams and Loading
    • 4.Amazon MSK (Kafka): Topics, Partitions, and When to Use What
    • 5.Week 2 Synthesis: Data Ingestion Part 1 Review
  • Week 3
    • 1.Streaming Processing: Managed Service for Apache Flink and Window Aggregation
    • 2.Ingestion Reliability: Idempotency, Ordering, Retries, Deduplication, DLQ
    • 3.CDC and Data Replication: Database Migration Service (DMS)
    • 4.Ingestion Architecture Patterns: Lambda Architecture and Event-Driven Ingestion
    • 5.Week 3 Synthesis: Data Ingestion Part 2 Review
  • Week 4
    • 1.Glue Data Catalog and Crawlers: Adding Metadata to Data
    • 2.Glue ETL Job: Transform Data on Spark
    • 3.Glue Studio and DataBrew: Transform Without Code
    • 4.Schema Management and Data Quality: Tolerate Evolution, Guarantee Trust
    • 5.Week 4 Synthesis: The Big Picture of AWS Glue Transformation
  • Week 5
    • 1.Amazon EMR: Spark, Hive and Cluster Operations, Plus EMR Serverless
    • 2.Lambda Transformation and Lightweight Processing: Event-Driven ETL's Limits and Fit
    • 3.Orchestration: Step Functions, MWAA, and Glue Workflows Selection Criteria
    • 4.Performance and Cost Optimization: File Format, Compression, Partitioning, and Small File Problem
    • 5.Week 5 Synthesis: Data Transformation 2 — Engines, Orchestration, Optimization Integration Review
  • Week 6
    • 1.S3 Data Lake Layout and Partitioning Strategy
    • 2.AWS Lake Formation Central Permission Management
    • 3.Open Table Formats: Iceberg, Hudi, Delta Lake
    • 4.S3 Storage Management and Cost Optimization
    • 5.Week 6 Comprehensive Review: Data Lake Recap
  • Week 7
    • 1.Amazon Redshift: Distribution and Sort Keys and Workload Optimization
    • 2.Amazon Athena: Serverless Queries and Cost Optimization
    • 3.DynamoDB (Analytics Perspective): Key Design and Stream-Based Pipelines
    • 4.RDS/Aurora and Store Selection: OLTP, Zero-ETL, Workload-to-Store Decision
    • 5.Week 7 Comprehensive Review: Analytics Stores Recap
  • Week 8
    • 1.Pipeline Monitoring: CloudWatch Metrics, Logs, and Alarms
    • 2.Data Quality and Validation: Glue Data Quality and Quality Gates
    • 3.Logging, Audit, and Troubleshooting: CloudTrail and Failure Recovery
    • 4.Cost and Performance Operations: Monitoring, Sizing, Auto Scaling
    • 5.Week 8 Comprehensive Review: Data Operations and Support Recap
  • Week 9
    • 1.Access Control: IAM and Lake Formation Permissions
    • 2.Encryption: KMS and Service-Specific Encryption
    • 3.Sensitive Data Protection: Macie and Masking
    • 4.Data Governance: Catalog, Lineage, Sharing, Auditing
    • 5.Week 9 Synthesis: Security and Governance Review
  • Week 10
    • 1.Integrated Review of Domains 1 & 2: Ingestion, Transformation & Storage Management
    • 2.Integrated Review of Domains 3 & 4: Operations & Support, Security & Governance
    • 3.Full-Length Practice Exam Pace: 8 Integrated Scenarios
    • 4.Common Traps & Keywords: "Requirement → Service" Translation Guide
    • 5.Final D-Day Prep: Exam Structure, Time Management & Scenario Breakdown Strategy
MLS-C01 · SpecialtyMachine Learning - Specialty
← DEA-C01/Week 3/Day 1
DEA-C01· AssociateWeek 3 · Day 1~17 min read

Day 1 - Streaming Processing: Managed Service for Apache Flink and Window Aggregation

Batch processing assumes "we'll process accumulated data later all at once." Streaming processing assumes "we process data the moment it arrives." The difference isn't just fast vs slow. Batch assumes data has boundaries (files, date partitions). Streams handle infinite unbounded data. To compute something like "today's revenue total" from endless data, you have to draw a boundary somewhere artificially. That boundary is called a window.

In AWS, Amazon Managed Service for Apache Flink handles this streaming aggregation. Formerly "Kinesis Data Analytics," it had SQL and Flink variants; the SQL version (KDA for SQL) stopped accepting new creations in 2023 and Apache Flink is now standard

The rest is Pro only

Week 1 is free for everyone. Week 2 onwards — plus mock exams and unlimited review — is included in the Pro plan.

See ProLogin
PreviousWeek 2 Synthesis: Data Ingestion Part 1 ReviewWeek 2 · Day 5Next Ingestion Reliability: Idempotency, Ordering, Retries, Deduplication, DLQWeek 3 · Day 2