Topic 15.6
Notification Systems and Distributed Job Schedulers
In one line
A notification system combines a durable event log (Kafka or Streams), per-user preferences and rate limits in Redis, deduplication, delayed delivery with sorted sets, and real-time push via Pub/Sub to WebSocket servers. A distributed scheduler combines sorted sets, streams, locks and idempotent workers.
Think of it like this
A post office that sorts mail (event log), respects "no junk mail" stickers (preferences), won't deliver more than a few letters a day to one house (rate limits), holds some letters until a requested date (delayed delivery), and rings the doorbell for urgent ones (real-time push).
Key ideas
- 01
Pipeline: producers emit notification requests to Kafka; workers check preferences (
HGET prefs:{user} email), rate limits (token bucket per user per channel), and dedup (SET notif:{user}:{dedupKey} NX EX 86400); schedule quiet-hours delivery in a sorted set; send via providers with idempotency keys. - 02
Real-time in-app: store the notification (database or
XADD inbox:{user} MAXLEN ~ 200) and publish to the WebSocket server holding the user; unread counts asHINCRBY. - 03
Scheduler: jobs in sorted sets scored by run time, claimed atomically into streams, processed by consumer groups with retries and DLQs (Topic 12.6); recurring jobs enqueued by a leader with a lease, or by an atomic per-occurrence claim.
- 04
Observability: queue lag, DLQ size, provider error rates, delivery latency percentiles.
Code & diagrams
Interview problem
The problem
Notification system
Design notifications for 50M users across push, email, SMS and in-app, with user preferences, no more than 3 marketing messages per day per user, quiet hours, no duplicates, and real-time in-app delivery. Where do Redis Pub/Sub, Streams and Kafka each fit?
Explain it without notes
Why fail closed for marketing limits but open for API rate limits?
Practice
Design the key for "max 3 marketing messages per user per local day" including time zones.
Trade-offs
- ↔
Kafka gives durable intake at scale; Redis gives fast per-user state and scheduling; Pub/Sub gives instant but lossy signals.
Done when you can
I can design a notification system that combines Kafka, Redis state, scheduling and Pub/Sub correctly.