System 12.3 — Notification System (HLD)
In one line
Fan-out, channel adapters, templates, batching and the provider-failure story — the LLD from Phase 4 at scale.
Think of it like this
The Phase 4 notification LLD scaled up to millions of users: same idea (channels, templates, retries), just now needing queues so a marketing blast to 50 million users doesn't overwhelm anything.
Key ideas
- 01
Flow: event source → template renderer → router (by user prefs) → channel adapters (email/SMS/push) → providers (SendGrid/Twilio/FCM) with retries.
- 02
Scale: 10M events/day → ~116/s; but bursts on campaigns 100x — the queue absorbs it.
- 03
Data: users (channels+preferences), templates, notifications (id, userId, channel, status), provider call statuses.
- 04
Build over buy decision: email → provider; SMS → provider; PUSH → FCM/APNs; in-app → your own store. Never build an email server.
- 05
Dedupe & throttle: per-user batching windows (same push collapsed), per-provider rate caps (SMS gateways throttle hard).
- 06
Failure: provider 5xx → retry w/ jittered backoff → DLQ; undeliverable (bounce) → suppress + mark user dormant.
- 07
Consistency: notifications are events, not state — eventual is fine; idempotent sends (notificationId) prevent duplicates.
- 08
Status tracking: queued → sending → sent/failed → the ops dashboard is a core feature.
Code & diagrams
One event in, many channels out — the queue is what absorbs a 5M-user campaign burst.
Explain without notes
A single campaign must reach 5M users in 10 minutes. Walk pipeline buffering, provider caps, and batch etiquette.
Practice
Design the per-user preference filter + the dedupe key; state the SLA on send-outage recovery.
Trade-offs
- ↔
Batching destroys latency for latency-critical alerts (OTP) → two pipelines: transaction vs campaign.
Run it in production
You've designed it. Now build, operate, and break the same idea hands-on in the DevOps courses:
Completion checklist
I can sketch notify-e2e (source→router→channel→provider→retry) with the provider-cap puzzle solved.