Unit 2.4 · Kafka and Messaging

RabbitMQ, SQS/SNS, Redis Streams: Choosing a Broker

Queues vs logs, RabbitMQ exchanges and acknowledgements, SQS visibility timeouts and DLQs, SNS fan-out, Redis Streams, and a decision guide.

Intermediate 45 min 2 lab steps 2 failure drills

Start here

The mental model

A QUEUE is a to-do list shared by workers: each task goes to one worker, and once done it's crossed off and gone. A LOG (Kafka) is a newspaper archive: everyone reads the same pages, at their own pace, and the pages stay. If you need 'do this job once', use a queue. If you need 'many teams react to what happened, and we might replay history', use a log.

Go deeper

How it works inside

01RabbitMQ

Producers publish to an EXCHANGE, which routes to QUEUES via BINDINGS. Exchange types: DIRECT (exact routing key), TOPIC (patterns like order.*.eu), FANOUT (every bound queue), HEADERS. Consumers receive from queues and ACK each message; unacked messages are redelivered if the consumer dies. prefetch (QoS) limits unacked messages per consumer, which is the main throughput and fairness knob.

Failed messages can be rejected to a DEAD LETTER EXCHANGE, and TTL plus DLX enables delayed retries. For durability use QUORUM QUEUES (Raft-replicated across 3 nodes; the old mirrored classic queues are removed in 4.x) and persistent messages with publisher confirms. RabbitMQ Streams add a Kafka-like log type when you need replay.

RabbitMQdiagram

Rendering diagram…

02SQS and SNS

SQS is a fully managed queue with no servers or partitions to think about. A received message becomes INVISIBLE for the VISIBILITY TIMEOUT; if the consumer deletes it in time, it's done, otherwise it reappears for another consumer. After maxReceiveCount failures, a REDRIVE POLICY moves it to a DLQ. STANDARD queues are nearly unlimited in throughput but at-least-once and best-effort ordered; FIFO queues give ordering per MESSAGE GROUP ID and deduplication, at lower throughput.

SNS is pub/sub: publish once, deliver to many subscribers (SQS queues, Lambda, HTTP). SNS → multiple SQS queues is the standard FAN-OUT pattern, each consumer service getting its own durable queue. EventBridge adds content-based routing rules and SaaS integrations (AWS course, SQS and SNS/EventBridge topics).

03Redis Streams and lighter options

Redis Streams (XADD, XREADGROUP, XACK) give a log with consumer groups inside Redis. They're good for modest volumes when Redis is already there, but data lives in memory and durability is Redis's (Part 3). Redis PUB/SUB is fire-and-forget: offline subscribers miss messages, so it's only for ephemeral notifications. NATS (with JetStream for persistence) is a lightweight, very fast option popular for internal platform messaging.

Do it

Hands-on lab

1
RabbitMQ: exchange, queues, and a DLX
Start RabbitMQ with the management UI (http://localhost:15672, guest/guest) and declare the topology with rabbitmqadmin (bundled in the management image).
terminal
$ docker run -d --name rabbit -p 5672:5672 -p 15672:15672 rabbitmq:4-management
docker exec rabbit rabbitmqadmin declare exchange name=orders type=topic
docker exec rabbit rabbitmqadmin declare exchange name=dlx type=fanout
docker exec rabbit rabbitmqadmin declare queue name=email.dead durable=true
docker exec rabbit rabbitmqadmin declare binding source=dlx destination=email.dead
docker exec rabbit rabbitmqadmin declare queue name=email durable=true arguments='{"x-queue-type":"quorum","x-dead-letter-exchange":"dlx","x-delivery-limit":3}'
docker exec rabbit rabbitmqadmin declare binding source=orders destination=email routing_key='order.placed.*'
docker exec rabbit rabbitmqadmin publish exchange=orders routing_key=order.placed.in payload='{"order":"o-1"}'
docker exec rabbit rabbitmqadmin list queues name messages
── expected output ──
+------------+----------+
| name | messages |
+------------+----------+
| email | 1 |
| email.dead | 0 |
+------------+----------+
2
SQS: visibility timeout and DLQ
Create a DLQ and a main queue that redrives after 3 receives. Receive a message without deleting it, and it comes back after the timeout.
terminal
$ aws sqs create-queue --queue-name email-dlq
# attrs.json: {"VisibilityTimeout":"30","RedrivePolicy":"{\"deadLetterTargetArn\":\"arn:aws:sqs:ap-south-1:123456789012:email-dlq\",\"maxReceiveCount\":\"3\"}"}
Q=$(aws sqs create-queue --queue-name email --attributes file://attrs.json --query QueueUrl --output text)
aws sqs send-message --queue-url $Q --message-body '{"order":"o-1"}'
aws sqs receive-message --queue-url $Q --query 'Messages[0].Body'
aws sqs receive-message --queue-url $Q --query 'Messages[0].Body' # immediately: invisible
── expected output ──
"{\"order\":\"o-1\"}"
null

Operate it

Knobs that matter

Setting	Default	What it does	When to change it
prefetch (RabbitMQ basic.qos)	unlimited	Max unacked messages per consumer.	Set 10–100: too high starves other consumers and uses memory; too low limits throughput.
x-delivery-limit (quorum queue)	20 (RabbitMQ 4.x)	Redeliveries before dead-lettering.	Lower (3–5) with a DLX so poison messages leave quickly.
VisibilityTimeout (SQS)	30s	How long a received message stays hidden.	Set above your worst-case processing time (Lambda: ≥ 6× function timeout).
maxReceiveCount (SQS redrive)	—	Receives before moving to the DLQ.	3–5 typically; alert on DLQ depth > 0.
ReceiveMessageWaitTimeSeconds (SQS)	0	Long polling duration.	20 to cut empty receives (cost) and latency.

3am practice

Failure drills

Each drill is a real failure mode. Read the scenario and the output, decide what's wrong, then reveal the diagnosis.

Drill #1

Same email sent twice

Some customers receive two receipts. The SQS worker takes ~40 s for large orders (PDF generation).

terminal

$ aws cloudwatch get-metric-statistics --namespace AWS/SQS --metric-name ApproximateAgeOfOldestMessage ... # and worker logs:

grep 'o-7781' worker.log

── what you'll see ──

09:14:02 worker-a received o-7781

09:14:32 worker-b received o-7781

09:14:41 worker-a sent receipt o-7781

09:15:10 worker-b sent receipt o-7781

Drill #2

RabbitMQ blocks all publishers

Consumers of one queue have been down for hours. Now every publisher in the system hangs, even for unrelated queues.

terminal

$ docker exec rabbit rabbitmq-diagnostics alarms

── what you'll see ──

Node rabbit@rabbit reported alarms: memory (resource_limit_alarm) — used 1.61 GiB, limit 1.6 GiB

Decide

Which broker?

Need	Kafka	RabbitMQ	SQS / SNS	Redis Streams
Model	Partitioned log, replay	Queues + routing	Managed queue / pub-sub	In-memory log
Throughput	Very high	High	Very high (standard)	Moderate–high
Ordering	Per partition	Per queue (single consumer)	FIFO per message group	Per stream
Replay history	Yes (retention)	Only with Streams	No	Yes (trimmed)
Routing	By topic/key only	Rich (exchanges)	SNS filter policies / EventBridge rules	Minimal
Ops burden	High (or MSK)	Medium	None	Low if Redis exists
Pick when	Event streaming, many consumers, CDC, analytics	Task queues, complex routing, RPC-style work	AWS-native apps, zero ops, fan-out	Light streaming alongside existing Redis

The bigger picture

Connects to

System Design · Kafka vs Traditional Queue

Kafka = replayable log; RabbitMQ = smart router.

System Design · Message Queue Basics

Producer → broker → consumer, and why the little words 'decouple', 'buffer', and 'retry' change everything.

AWS · SQS

Deeper dive on queues, FIFO, and Lambda triggers.

AWS · SNS & EventBridge

Fan-out and event routing on AWS.

Part 3 · Redis

Redis's durability model, which Redis Streams inherit.

Prove it

Interview questions

Kafka or RabbitMQ: how do you choose?

What's the visibility timeout in SQS and how does it cause duplicates?

0/4 · 0%

Command Palette