Kafka vs Redis, RabbitMQ, SQS, and Pub/Sub

In one line

Choose messaging technology from requirements, not habit: Kafka for durable, replayable, high-throughput, ordered-per-key streams with many consumers; RabbitMQ for flexible routing and per-message work queues; SQS and Google Pub/Sub for fully managed queues and fan-out; Redis for sub-millisecond state and lightweight streams.

0/7 · 0%

Think of it like this

Choosing transport. A freight train (Kafka) moves enormous volumes on fixed tracks and keeps a manifest of everything; a courier network (RabbitMQ) routes individual parcels flexibly; a postal service you don't run yourself (SQS, Pub/Sub) just works; a motorbike messenger (Redis) is fastest across town but carries little and keeps no records.

Key ideas

01
Kafka: log with retention and replay, consumer groups with independent offsets, per-partition order, very high throughput, strong durability; you plan partitions and operate a cluster (or buy a managed one).
02
RabbitMQ: exchanges and bindings for rich routing (topic, headers, fan-out), per-message acks, priorities, delayed messages via plugins, DLX built in; messages are gone once consumed (Streams in RabbitMQ add log-style queues); moderate throughput.
03
Amazon SQS: managed queue with visibility timeouts, DLQs, FIFO queues (ordering per message group, lower throughput), no replay; SNS for fan-out. Google Pub/Sub: managed pub/sub with acks, ordering keys, message retention and seek (limited replay).
04
Redis: in-memory structures, Pub/Sub (at-most-once), Streams (consumer groups, bounded by RAM); ideal for caches, counters, rate limiters and low-latency coordination, not a long-lived event log.
05
Dimensions to compare aloud: durability, latency, replay, ordering scope, throughput, consumer model, routing, retention, operational cost, cloud integration.

Code & diagrams

comparison.txttext

                 Kafka              RabbitMQ            SQS / SNS          Google Pub/Sub       Redis
Model            partitioned log    broker queues       managed queue      managed pub/sub      in-memory structures
Replay           yes (retention)    no (streams: yes)   no                 limited (seek)       Streams only, RAM-bound
Ordering         per partition      per queue           FIFO per group     per ordering key     per stream / list
Throughput       very high          medium              high (std)         high                 very high, small data
Consumer model   groups + offsets   competing + acks    visibility timeout acks + subscriptions groups (Streams)
Routing          by topic/key       rich exchanges      SNS filters        filters              manual
Retention        hours-forever      until consumed      up to 14 days      up to 31 days        RAM + TTL
Ops              heavy (or managed) medium              none               none                 light

Interview problem

The problem

Kafka, RabbitMQ, SQS or Redis?

Choose and justify a technology for: (1) order events consumed by 8 services with replay needs, (2) background thumbnail jobs with retries and priorities, (3) a serverless app on AWS needing simple decoupling, (4) real-time leaderboard updates and rate limiting.

Explain it without notes

When is RabbitMQ a better fit than Kafka?

Practice

Classify five messaging needs in your organisation by the comparison dimensions and pick a technology for each.

Trade-offs

↔
Kafka's power comes with operational weight; managed queues trade features for simplicity.

Done when you can

I can compare Kafka with RabbitMQ, SQS, Pub/Sub and Redis on concrete dimensions.

Command Palette