Day 3 - Kinesis: A Log That Holds On to Flowing Data

SQS and SNS treat messages as things that "disappear once processed." When a consumer takes a message, it is deleted from the queue; once published, a message is pushed out and forgotten. Some data is fundamentally different, though. User click streams, IoT sensor readings, and application logs flow continuously, multiple systems want to read the same data at the same time for different purposes, and sometimes you even want to reprocess yesterday's data today with a new algorithm. For this kind of streaming data, Amazon Kinesis took a completely different model: instead of discarding messages after consumption, it only appends them to a log and retains them for a set period.

In DVA-C02, Kinesis tends to show up in comparison scenarios asking "how is it different from SQS and SNS?" Details such as shard throughput numbers, the division of roles between Data Streams and Firehose, and Enhanced Fan-Out are also in scope. This article covers why Kinesis chose the "append-only log" structure, how shards and partition keys solve ordering and scaling at the same time, why Data Streams and Firehose diverge, and where Kinesis is fundamentally different from SQS.

The Append-Only Log: The Heart of Kinesis

If you boil the Kinesis Data Streams data model down to one word, it is log. It is not a database or a queue; it is an immutable sequence to which records are only appended at the end in time order, with nothing in the middle ever modified. Producers attach records to the end of the log, and consumers start reading from any point in the log they want (a sequence number). Reading does not make the data disappear; it stays in place for the retention period (24 hours by default, up to 365 days).

This simple structure makes three things possible that SQS cannot do.

• Multiple independent consumers: Several consumers read the same stream at the same time, each at its own position. A follows the end of the stream for real-time alerts, while B reads data from an hour ago for batch analysis. They do not interfere with each other.
• Reprocessing (replay): You can rerun a new consumer with a bug fix starting from yesterday to reprocess past data. With SQS this is impossible, because messages disappear once consumed.
• Order preservation: A log is inherently an ordered sequence, so data within the same partition is read in the order it was written.

The idea of "putting the log at the center of a data system" is not unique to Kinesis; it is a major trend in distributed systems. Jay Kreps (co-creator of Kafka) laid out this thinking in his article "The Log: What every software engineer should know about real-time data's unifying abstraction" (2013). His point was that database replication, messaging, and event sourcing all ultimately unify into a single abstraction: the "append-only log." The root of this idea is the database WAL (Write-Ahead Log). A DB records changes in the log first, before writing them to the data files, and with that log alone any state can be reconstructed. Kinesis and Kafka turned this insight, "the log is the source of truth," into a messaging system itself.

The difference between Kinesis and SQS comes down to "does consumption destroy the data?"

• In SQS, consumption is deletion via DeleteMessage (destructive read).
• In Kinesis, reading and retention are separate, so the consumer has to remember for itself "how far have I read" (checkpoint).
• This is why KCL (Kinesis Client Library) stores that checkpoint in a DynamoDB table for you. The log itself does not know the consumer's position, so the responsibility for tracking position lies with the consumer.
• This is the same structure as Kafka's consumer offsets. "The stream preserves the truth, and the consumer is responsible for how far it has read" is the core contract of log-based systems.

Shards and Partition Keys: Ordering and Scaling at the Same Time

If you keep a single log as just one line, throughput is tied to the ceiling of that one line (the same problem we saw with SQS FIFO). Kinesis splits a stream into multiple shards to parallelize. Each shard is an independent log with its own throughput limits.

The key here is the partition key. When a producer specifies a partition key while sending a record, Kinesis hashes that key to decide which shard it goes to. Records with the same partition key always go to the same shard, and order is preserved within a single shard. So "if you use the order ID as the partition key, events for the same order are processed in order, while different orders are spread across multiple shards and processed in parallel." It is the same idea as MessageGroupId in SQS FIFO.

import boto3, json
kinesis = boto3.client('kinesis')
 
kinesis.put_record(
    StreamName='clickstream',
    Data=json.dumps({'userId': 'U123', 'page': '/checkout'}),
    PartitionKey='U123'   # This user's events go to the same shard → order preserved
)

⚠️ Trap: Poor partition key design creates a hot shard. For example, if you give every record the same partition key, they all pile onto one shard, and even though the other shards sit idle, that one shard is capped at 1 MB/s and raises ProvisionedThroughputExceededException. On the exam, the symptom "only a specific shard is throttled" is a clue pointing to a skewed partition key. The fix is to design keys so they are spread evenly (high cardinality). It is the same structure as DynamoDB's hot partition problem; both are common pitfalls of "systems that distribute by key hash."

With classic reads, "all consumers share 2 MB/s." With 3 consumers, each can read only about 0.67 MB/s, and since they all poll GetRecords, latency stretches from 200 ms to several seconds. Enhanced Fan-Out (EFO) gives each consumer a dedicated 2 MB/s pipe via HTTP/2 push. Even as consumers increase, they don't take throughput from one another, and latency drops to about 70 ms. The cost is money (per consumer-shard hour plus data transfer). The answer to a scenario like "multiple consumers need to read the same stream with low latency" is EFO. The shift from pull (classic) to push (EFO) is in the same vein as SQS polling versus SNS push.

Data Streams vs. Firehose: Process It, or Deliver It

Kinesis is not a single service but a family. The distinction that matters on the exam is between Data Streams and Data Firehose.

Data Streams is the shard-based log we just looked at. Consumers (Lambda, KCL apps, EFO consumers) read the data directly and process it in real time; use it when you need reprocessing, multiple consumers, and low latency. You either manage shards yourself (Provisioned) or let On-Demand scale automatically.

Firehose has a completely different purpose. It is a serverless ETL pipeline that buffers data and automatically delivers it to a destination. There is no consumer code and no shard management. It streams data to S3, Redshift, OpenSearch, and Splunk, and along the way it can transform data with Lambda or convert JSON to Parquet format.

The deciding criterion is "do I need to process the data with my own code, or do I just need to send it to S3/Redshift?" For the former, use Data Streams; for the latter, Firehose.

[Click events / IoT / logs]
        │
        ▼
[Kinesis Data Streams] ──> Lambda (real-time alerts, anomaly detection)
        │              └──> KCL app (complex stateful processing)
        ▼
[Kinesis Data Firehose] ──buffering (60s/1MB)──> S3 (data lake, Parquet)
                                          └──> Redshift / OpenSearch

There is a configuration that uses the two services together. Real-time analytics is processed immediately with Data Streams + Lambda, while at the same time Firehose subscribes to the same stream and archives all the raw data to S3 (to be analyzed later with Athena). Thanks to the "multiple independent consumers" we saw earlier, the real-time consumer and the archive consumer read one stream at the same time without interfering with each other. Streaming CloudWatch Logs to Firehose through a Subscription Filter for long-term retention in S3 is the same ETL pattern.

⚠️ Trap: Don't be fooled by the word "real-time." Firehose has at least 60 seconds of buffering, so it is not truly real-time (Near Real-Time). If you need "immediate processing within 1 second," you must consume Data Streams directly, and Firehose is for cases where "we're going to collect it in S3 for batch analysis anyway, so a 1-minute delay is fine." On the exam, "ultra-low-latency real-time processing" means Data Streams, and "automatic loading into S3 (serverless, minimal code)" means Firehose.

Kinesis vs. SQS vs. SNS: When to Use What

Choosing among these three services is a topic that is easy to ask as a scenario question. On the surface they all seem to "move messages," but their models are fundamentally different.

To restate the essential difference between Kinesis and SQS:

• SQS consumes and deletes per message, and the consumer tracks "have I processed each message?" (visibility timeout).
• Kinesis reads in order per shard, and the consumer tracks "how far have I read?" (checkpoint).
• In SQS, even if one message is poison, you just pull that one out to a DLQ. In Kinesis, if one record fails processing, the records behind it get blocked (the price of preserving order).
• To deal with this "head-of-line blocking," the Lambda-Kinesis integration provides options such as BisectBatchOnFunctionError, MaximumRetryAttempts, and DestinationConfig (sending failed records to SQS/SNS). The difference in model translates directly into a difference in failure handling.

Summary

The one sentence that runs through Kinesis is: "Instead of discarding messages after consumption, retain them in an append-only log so multiple consumers can read them repeatedly, each at its own position." This log model makes possible the multiple independent consumers, reprocessing, and order preservation that SQS cannot offer, and in exchange the consumer is responsible for its own position (checkpoint). Shards parallelize the log and partition keys solve ordering and distribution together (hot shards are the pitfall). Roles split between Data Streams for direct processing and Firehose for automatic delivery, and using both together gets you real-time processing and archiving from one stream. The choice among SQS, SNS, and Kinesis boils down to a difference in model: "does consumption destroy the data (SQS), does it push 1:N (SNS), or does it allow multiple consumers and reprocessing over a retained log (Kinesis)?"

In the next article, we move on to Step Functions, which weaves these individual services into a single flow as orchestration, and AppSync, which ties multiple data sources together behind a single GraphQL API.