Topic 8.3
Streaming & Orchestration: Kinesis and Step Functions
In one line
Kinesis Data Streams handles ordered, replayable real-time event streams (AWS's managed alternative to Kafka for many cases); Firehose delivers streams into S3 and analytics stores; Step Functions orchestrates multi-step workflows with retries and state.
Think of it like this
A sushi conveyor belt (Kinesis): plates keep moving in order, several diners can take from it, and it keeps running at its own pace. Step Functions is the head chef's order ticket: 'prepare, then cook, if it burns start again, then plate, then serve', with each step tracked.
Key ideas
- 01
KINESIS DATA STREAMS: records go to SHARDS (ordered by partition key, like Kafka partitions: Stateful Systems course, Kafka), retained 24 hours to 365 days, read by multiple consumers (Lambda, KCL apps, Flink). On-demand mode scales automatically; provisioned mode charges per shard. Choose it over SQS when you need ordering per key, replay, and multiple independent consumers; choose MSK when you need the Kafka API or ecosystem.
- 02
AMAZON DATA FIREHOSE: fully managed delivery of streaming data to S3, Redshift, OpenSearch, or HTTP endpoints, with buffering, compression, format conversion (JSON → Parquet), and optional Lambda transforms. The easy path for clickstreams and logs into a data lake (Stateful Systems course, batch vs stream).
- 03
STEP FUNCTIONS: state machines (in Amazon States Language) that call Lambda and 200+ AWS services directly: Task, Choice, Parallel, Map, Wait states, with built-in RETRY/CATCH, timeouts, and a visual execution history. STANDARD workflows run up to a year with exactly-once steps (orders, approvals, sagas: System Design course, saga); EXPRESS workflows are cheap, high-volume, short-lived (event processing).
- 04
EVENTBRIDGE PIPES and SCHEDULER connect sources (SQS, Kinesis, DynamoDB streams) to targets with filtering and enrichment without glue code (Phase 6, SNS and EventBridge).
Code & diagrams
aws kinesis create-stream --stream-name clicks --stream-mode-details StreamMode=ON_DEMAND
aws kinesis put-record --stream-name clicks --partition-key user-42 \
--data "$(echo -n '{"page":"/p/42","ts":1790487120}' | base64)"
SHARD=$(aws kinesis list-shards --stream-name clicks --query 'Shards[0].ShardId' --output text)
IT=$(aws kinesis get-shard-iterator --stream-name clicks --shard-id $SHARD --shard-iterator-type TRIM_HORIZON --query ShardIterator --output text)
aws kinesis get-records --shard-iterator $IT --query 'Records[].Data' --output text | base64 -dExplain it without notes
When would you choose Kinesis Data Streams over SQS?
Practice
Design an order-fulfilment workflow that retries payment 3 times, waits up to 24 hours for warehouse confirmation, and compensates on failure.
Trade-offs
- ↔
Orchestration (Step Functions) makes workflows visible and centrally managed; choreography (events between services) is more decoupled but harder to trace. Kinesis vs MSK trades AWS-native simplicity for the Kafka ecosystem.
Done when you can
I can choose between SQS, SNS, EventBridge, Kinesis, and MSK
I can design a Step Functions workflow with retries and compensation