Yesterday's batch ingestion was "collect then process all at once." Starting today we enter the opposite world — streaming. Clickstreams, IoT sensors, payment transactions, and game telemetry all represent data flowing endlessly in, needing sub-second processing. At the heart of this is Amazon Kinesis Data Streams (KDS).
KDS is, in one sentence, a "scalable real-time data pipe." Many producers push data into it, many consumers read it away almost simultaneously. Today we examine the internals of this pipe (shards), how to decide which partition gets a record (partition keys), and throughput calculations.