Day 5 - Week 2 Synthesis: Data Ingestion Part 1 Review

📌 Key Takeaways

• Ingestion questions narrow to a single service if you work in this order: source (files or events?) → direction (pull or receive?) → line and scale → freshness → operating conditions.
• The four batch options: small and one-off = S3 upload, large, recurring, verified = DataSync, an external party pushes via SFTP = Transfer Family, insufficient line or huge volume = Snow.
• The three streaming options: multiple consumers and replay = KDS, loading without code = Firehose, existing Kafka assets = MSK.
• "Real-time analytics + raw data retention" is the combination where KDS (or MSK) is the source and Firehose loads into S3.
• Traps usually come from limits and defaults: parallelism ceiling = number of shards/partitions, Firehose has no retention or replay, Redshift loading is COPY via S3.

The Ingestion Service Map on One Page

This week we surveyed "ingestion," the starting point of data engineering, divided into batch and streaming. Today we tie the scattered pieces into a single decision map. Ninety percent of ingestion-domain questions on DEA-C01 are "which service should I pick for this scenario," and the answer always comes from the keywords. The goal of this synthesis is to make the scenario → service mapping come to mind reflexively.

                        Data ingestion
                            │
        ┌───────────────────┴───────────────────┐
      Batch                                  Streaming
        │                                       │
  ┌─────┼─────────┬──────────┐         ┌────────┼──────────┬─────────┐
S3 upload DataSync  Transfer   Snow     KDS      Firehose    MSK
(one-off/ (recurring/ Family  (PB/slow (real-time (near-real-  (existing
 small)    large,    (receive  line)    multi-     time,       Kafka)
           sync &    SFTP)              consumer   simple
           verify)                      · replay)  loading)

Batch Four Review

💡 Related theory: The two axes for choosing batch are "data volume" and "line situation." If the line is sufficient, go online (DataSync); if it is insufficient or the data is too large, go offline (Snow). And a direction axis is added: "do I pull it in (DataSync) vs does an external party push it in (Transfer Family)?" If you hold just these two axes, nearly every batch question can be solved.

Streaming Three Review

KDS      : Real-time pipe. Per-shard. Multiple consumers and replay possible. You write the consumer code yourself.
Firehose : Near-real-time loading pipe. After buffering, loads automatically into S3/Redshift/OpenSearch. Almost no code.
MSK      : Managed Apache Kafka. Standard open source, portability. Best for migrating existing Kafka assets.

Let us nail down the three key distinctions again.

1. KDS vs Firehose: For complex real-time processing, multiple consumers, or replay, use KDS; for simple loading, use Firehose. The combination that chains them (KDS source → Firehose loads into S3) is the answer for "real-time analytics + raw data retention" scenarios.
2. Kinesis vs MSK: If you see "existing Apache Kafka," MSK; if "minimal operations, AWS-native, simple loading," Kinesis.
3. Common log model: KDS and Kafka are both append-only logs, so data does not disappear when consumed, and multiple consumers and replay work. This differs from SQS.

# Combined pattern: run KDS (real-time analytics) + Firehose (raw archive to S3) together
import boto3
firehose = boto3.client("firehose")
 
firehose.create_delivery_stream(
    DeliveryStreamName="raw-archive",
    DeliveryStreamType="KinesisStreamAsSource",          # specify KDS as the source
    KinesisStreamSourceConfiguration={
        "KinesisStreamARN": "arn:aws:kinesis:...:stream/clickstream",
        "RoleARN": "arn:aws:iam::123456789012:role/firehose-role"
    },
    S3DestinationConfiguration={
        "BucketARN": "arn:aws:s3:::my-data-lake",
        "RoleARN": "arn:aws:iam::123456789012:role/firehose-role",
        "BufferingHints": {"SizeInMBs": 128, "IntervalInSeconds": 300},
        "CompressionFormat": "GZIP"
    }
)
# A real-time analytics Lambda consumes the same clickstream separately -> analytics + archiving achieved together

A well-designed streaming architecture loosely couples "ingest → process → load." KDS/MSK act as the ingestion and buffer layer while feeding several downstreams at once (real-time analytics, data lake loading, search indexing). Thanks to this separation, even if one consumer slows down, other consumers and producers are not blocked. This is why a log-based stream suits a data platform better than a simple queue.

Commonly Missed Pitfalls

• Incomplete multipart pieces → invisible S3 cost, solved with a Lifecycle abort rule
• Hot shard/hot partition → skewed partition key, replace with a high-cardinality key
• Small files problem → increase the Firehose buffer size to load larger files
• Firehose cannot replay → if you need replay or multiple consumers, KDS
• Parallelism ceiling = number of shards/partitions → adding only consumers does no good
• Redshift loading is COPY via S3 → not a single-row INSERT

Decision Order: Ask Five Times and One Remains

① What is coming in?
   ├─ Chunks of files/objects ─────────► Batch side (go to ②)
   └─ Small records generated continuously ► Streaming side (go to ④)

② Who moves it?
   ├─ An external partner uploads via SFTP/FTPS ► Transfer Family
   └─ We fetch it (go to ③)

③ What are the line and scale?
   ├─ Small, one-off ──────────────────► S3 upload (CLI/SDK, multipart for large files)
   ├─ Large, recurring, sufficient line ► DataSync
   └─ Insufficient line, or huge and one-time ► Snow family

④ Are there existing assets?
   ├─ Kafka code, connectors, team ────► MSK (MSK Serverless if you also dislike managing capacity)
   └─ None (go to ⑤)

⑤ What does the reading side require?
   ├─ Just accumulate in a destination ► Data Firehose
   ├─ Multiple apps each read / replay ► Kinesis Data Streams
   └─ Both ────────────────────────────► KDS + Firehose (with KDS as the source)

• Skipping ① and looking at ⑤ first leads to errors. The mistake of picking Firehose for a "sync a file server every night" question comes from here.
• ④ comes before ⑤. If "existing Apache Kafka" is in the passage, the MSK family comes first regardless of what the reading side requires.

Detailed Comparison of the Three Streaming Services

We put the last three days' content on the same axes in one table. Reading across a row shows the differences.

• The three services are both substitutes and combinable pieces. KDS → Firehose and MSK → Firehose are both official configurations. When a combination appears in the answer choices, judge it by "do the roles each one takes overlap?"

Week 2 Numbers and Defaults on One Page

• Most limits are adjustable defaults and differ by Region. What the exam asks about is less the number itself than "which limit do you hit first." For example, when records are small, the record-count limit is hit before the byte limit, and when there are many consumers, the read limit fills before the write limit.

Scenario Practice: From Clue to Answer

These situations do not overlap with the practice questions. Cover the decisive clue column and answer first.

Symptom → Cause → Action

⚠️ Pitfall: Fewer of the actions in the table than you would expect have "increase shards" as the answer. Skew is a key problem, read contention is a fan-out problem, and a blockage is an error-handling problem. Capacity increase is the right remedy only "when all shards evenly hit the limit."

Week 3 moves on to the other half of ingestion: real-time processing and windows on top of streams, ingestion reliability (idempotency, retries, DLQ), DMS and CDC, and event-driven architecture. On top of the "what do we bring it in with" map we built today, "how do we handle the incoming flow safely" is layered. Check the keyword mapping for the ingestion domain on this map.

📖 Glossary

• Ingestion: The stage of bringing data from the source into the analytics environment. It divides into batch and streaming.
• Incremental sync: A method that picks and moves only what changed since the last run. DataSync's scheduled tasks are the typical example.
• Append-only log: A record structure that only adds to the end. Reading does not erase it, so multiple consumers can re-read from their own positions.
• Fan-out: Multiple consumers each receiving one stream independently.
• Replay: Re-reading and processing past records within the retention period from the beginning.
• Hot shard / hot partition: The phenomenon where keys are skewed and load concentrates only on a particular shard or partition.
• Poison pill record: A record that fails every time it is processed and blocks the processing of the records behind it.
• Staging: An intermediate storage location passed through briefly before entering the final destination. Firehose's Redshift loading uses S3 as staging.
• At-least-once: A delivery guarantee that delivers at least once but may produce duplicates. The receiving side is designed to tolerate duplicates.