Cert Notes/ Commute Study Notes
Roadmap
KOEN
CLF-C02 · FoundationalCloud Practitioner - Foundational
DVA-C02 · AssociateDeveloper - Associate
SAA-C03 · AssociateSolutions Architect - Associate
SOA-C02 · AssociateCloudOps Engineer - Associate
SAP-C02 · ProfessionalSolutions Architect - Professional
DOP-C02 · ProfessionalDevOps Engineer - Professional
SCS-C03 · SpecialtySecurity - Specialty
MLA-C01 · AssociateMachine Learning Engineer - Associate
AIF-C01 · FoundationalAI Practitioner - Foundational
DEA-C01 · AssociateData Engineer - Associate
  • Week 1
    • 1.What Is Data Engineering
    • 2.Batch vs Streaming
    • 3.A Bird's-Eye View of AWS Data Services
    • 4.Data Formats and Modeling
    • 5.Week 1 Comprehensive Review
  • Week 2
    • 1.Batch Ingestion: S3 Upload, DataSync, Transfer Family, Snow
    • 2.Kinesis Data Streams: Shards, Partition Keys, and Throughput
    • 3.Kinesis Data Firehose: Delivery Streams and Loading
    • 4.Amazon MSK (Kafka): Topics, Partitions, and When to Use What
    • 5.Week 2 Synthesis: Data Ingestion Part 1 Review
  • Week 3
    • 1.Streaming Processing: Managed Service for Apache Flink and Window Aggregation
    • 2.Ingestion Reliability: Idempotency, Ordering, Retries, Deduplication, DLQ
    • 3.CDC and Data Replication: Database Migration Service (DMS)
    • 4.Ingestion Architecture Patterns: Lambda Architecture and Event-Driven Ingestion
    • 5.Week 3 Synthesis: Data Ingestion Part 2 Review
  • Week 4
    • 1.Glue Data Catalog and Crawlers: Adding Metadata to Data
    • 2.Glue ETL Job: Transform Data on Spark
    • 3.Glue Studio and DataBrew: Transform Without Code
    • 4.Schema Management and Data Quality: Tolerate Evolution, Guarantee Trust
    • 5.Week 4 Synthesis: The Big Picture of AWS Glue Transformation
  • Week 5
    • 1.Amazon EMR: Spark, Hive and Cluster Operations, Plus EMR Serverless
    • 2.Lambda Transformation and Lightweight Processing: Event-Driven ETL's Limits and Fit
    • 3.Orchestration: Step Functions, MWAA, and Glue Workflows Selection Criteria
    • 4.Performance and Cost Optimization: File Format, Compression, Partitioning, and Small File Problem
    • 5.Week 5 Synthesis: Data Transformation 2 — Engines, Orchestration, Optimization Integration Review
  • Week 6
    • 1.S3 Data Lake Layout and Partitioning Strategy
    • 2.AWS Lake Formation Central Permission Management
    • 3.Open Table Formats: Iceberg, Hudi, Delta Lake
    • 4.S3 Storage Management and Cost Optimization
    • 5.Week 6 Comprehensive Review: Data Lake Recap
  • Week 7
    • 1.Amazon Redshift: Distribution and Sort Keys and Workload Optimization
    • 2.Amazon Athena: Serverless Queries and Cost Optimization
    • 3.DynamoDB (Analytics Perspective): Key Design and Stream-Based Pipelines
    • 4.RDS/Aurora and Store Selection: OLTP, Zero-ETL, Workload-to-Store Decision
    • 5.Week 7 Comprehensive Review: Analytics Stores Recap
  • Week 8
    • 1.Pipeline Monitoring: CloudWatch Metrics, Logs, and Alarms
    • 2.Data Quality and Validation: Glue Data Quality and Quality Gates
    • 3.Logging, Audit, and Troubleshooting: CloudTrail and Failure Recovery
    • 4.Cost and Performance Operations: Monitoring, Sizing, Auto Scaling
    • 5.Week 8 Comprehensive Review: Data Operations and Support Recap
  • Week 9
    • 1.Access Control: IAM and Lake Formation Permissions
    • 2.Encryption: KMS and Service-Specific Encryption
    • 3.Sensitive Data Protection: Macie and Masking
    • 4.Data Governance: Catalog, Lineage, Sharing, Auditing
    • 5.Week 9 Synthesis: Security and Governance Review
  • Week 10
    • 1.Integrated Review of Domains 1 & 2: Ingestion, Transformation & Storage Management
    • 2.Integrated Review of Domains 3 & 4: Operations & Support, Security & Governance
    • 3.Full-Length Practice Exam Pace: 8 Integrated Scenarios
    • 4.Common Traps & Keywords: "Requirement → Service" Translation Guide
    • 5.Final D-Day Prep: Exam Structure, Time Management & Scenario Breakdown Strategy
MLS-C01 · SpecialtyMachine Learning - Specialty
← DEA-C01/Week 2/Day 4
DEA-C01· AssociateWeek 2 · Day 4~15 min read

Day 4 - Amazon MSK (Kafka): Topics, Partitions, and When to Use What

The final piece of streaming ingestion is Amazon MSK (Managed Streaming for Apache Kafka). If Kinesis is AWS's proprietary streaming service, Kafka is the industry-standard open-source streaming platform, and MSK is AWS offering Kafka as a managed service.

The key question mirrors yesterday's: "We have Kinesis, so why MSK? When do we use what?" Today we understand Kafka's fundamentals (topics, partitions, consumer groups) and see exactly where they diverge from Kinesis. The exam almost always compares these two.

Kafka's Structure: Topics and Partitions

In Kafka, data is organized into topics — think of them as "subject-based logs" (e.g., orders, clicks, sensor-data)

The rest is Pro only

Week 1 is free for everyone. Week 2 onwards — plus mock exams and unlimited review — is included in the Pro plan.

See ProLogin
PreviousKinesis Data Firehose: Delivery Streams and LoadingWeek 2 · Day 3Next Week 2 Synthesis: Data Ingestion Part 1 ReviewWeek 2 · Day 5