Choosing a Queue: Lists, Streams, Pub/Sub, Kafka, RabbitMQ

In one line

Pick a messaging tool by its guarantees: whether messages persist, whether they're acknowledged, whether they can be replayed, how consumers scale, and how much operational weight you can carry. Redis covers low-latency and moderate-durability cases; Kafka and RabbitMQ cover durable logs and rich routing.

0/5 · 0%

Think of it like this

Sending things across town. A shout across the street (Pub/Sub) is instant but nobody hears it if they're inside. A to-do pile on a desk (list) works for one office. A signed-for courier with a delivery log (Streams) handles retries. A national postal system with warehouses (Kafka) keeps everything for weeks and serves thousands of recipients, at much higher cost.

Key ideas

01
Dimensions to compare: persistence, acknowledgement, replay (can a new consumer read history?), consumer groups (load balancing), ordering guarantees, backpressure (what happens when consumers are slow), retries and dead letters, retention size (RAM vs disk), throughput, and operational complexity.
02
Redis lists: simple FIFO, persistence via Redis, no acks (hand-built with LMOVE), no replay, one consumer per message. Good for simple background jobs in an app that already uses Redis.
03
Redis Streams: persisted log, consumer groups, acks, PEL, claiming, replay by ID, bounded by memory. Good for event distribution and work queues up to moderate volumes and retention (hours to days).
04
Redis Pub/Sub: no persistence, no acks, fan-out to connected subscribers only. Good for live signals and invalidation broadcasts.
05
Kafka: disk-based replicated log with configurable acks (acks=all, min.insync.replicas), partitions for ordered parallelism, retention for days to forever, replay from any offset, huge throughput, and a heavier operational footprint. RabbitMQ: flexible routing (exchanges, bindings), per-message acks, DLQs built in, priority queues. SQS: fully managed queue with visibility timeouts and DLQs, no ops.
06
A common architecture uses several: Kafka as the durable event backbone, Redis Streams or lists for fast local work queues, Pub/Sub for real-time pushes.

Code & diagrams

comparison.txttext

                 Redis List    Redis Stream     Redis Pub/Sub   Kafka             RabbitMQ
Persistence      Redis RDB/AOF Redis RDB/AOF    none            disk, replicated  disk (durable q)
Acknowledgement  hand-built    XACK + PEL       none            offsets commit    per-message ack
Replay           no            yes (by ID)      no              yes (by offset)   no (once consumed)
Consumer groups  no            yes              no (fan-out)    yes               competing consumers
Ordering         FIFO          per stream       per channel     per partition     per queue
Retention        RAM           RAM (trim)       none            disk, days-forever until consumed
Backpressure     list grows    stream grows     drops slow subs consumer lag      queue grows / flow control
DLQ              hand-built    hand-built       n/a             hand-built/tools  built in
Latency          sub-ms        sub-ms           sub-ms          ms                ms
Ops weight       tiny          small            tiny            heavy             medium

Interview problem

The problem

Order events: Redis or Kafka?

The order service emits events consumed by email, analytics, fraud, and a search indexer. Some consumers are added later and need history. Decide what goes to Redis and what goes to Kafka, and justify it with latency, durability, replay, ordering, throughput, consumer model, operations and retention.

You're given

5K events/sec, bursts to 50K
New consumers need 30 days of history
Fraud checks need < 50 ms
Events must never be lost

The interviewer follows up

When would Redis Streams alone be enough?

Explain it without notes

List six dimensions you use to compare messaging systems.

Practice

Choose a tool for: (a) resize uploaded images, (b) push "user is typing", (c) feed a data warehouse with every click, (d) invalidate a product cache on 50 app servers.

Trade-offs

↔
Redis messaging is fast and simple to run but bounded by RAM and Redis durability; Kafka is durable and scalable but heavier to operate.
↔
Using several tools gives each workload the right guarantees at the cost of more moving parts.

Done when you can

I can compare lists, Streams, Pub/Sub, Kafka and RabbitMQ on concrete dimensions.
I can split a real event architecture between Redis and Kafka and justify it.

Command Palette