Topic 15.3
Feeds and Timelines: Instagram and Twitter
In one line
Timelines are built by fan-out on write (push post IDs into followers' timeline caches), fan-out on read (merge followees' recent posts at read time), or a hybrid that pushes for normal users and pulls for celebrities. Redis holds per-user timeline caches as capped sorted sets or lists of post IDs.
Think of it like this
Newspapers. Delivering a copy to every subscriber's door each morning is fan-out on write (fast to read, lots of delivery work). Making readers visit the newsstand and assemble their own selection is fan-out on read. For a celebrity columnist with 100M readers, you don't print 100M personal copies; readers pick them up.
Key ideas
- 01
Fan-out on write: when user A posts, a worker adds the post ID to each follower's timeline:
ZADD timeline:{follower} <ts> <postId>thenZREMRANGEBYRANK timeline:{follower} 0 -801to keep the latest 800. Reads are oneZRANGE ... REV LIMITplus a hydration step (MGET/HMGETpost details from a cache). - 02
The celebrity problem: a user with 50M followers would cause 50M writes per post, delaying delivery for everyone. Hybrid: skip fan-out for accounts above a follower threshold; at read time, merge the user's precomputed timeline with recent posts from the few celebrities they follow (
ZRANGE posts:{celebrity}). - 03
Fan-out on read only: store per-author post lists; read merges N followees' lists. Cheap writes, expensive reads for users following thousands; fine for small graphs.
- 04
Memory: 200M active users × 800 IDs × ~16 bytes of listpack or ~60 bytes of skiplist entries is large; keep timelines only for recently active users (TTL of a few days), and rebuild on demand for returning users.
- 05
Kafka carries post-created events to fan-out workers; Redis holds the materialised timelines; the database or a wide-column store holds the posts and the social graph.
Code & diagrams
ZADD timeline:{u42} 1727520000 post:9001
ZREMRANGEBYRANK timeline:{u42} 0 -801 # keep newest 800
EXPIRE timeline:{u42} 604800
ZRANGE timeline:{u42} +inf 1727519000 BYSCORE REV LIMIT 0 20 # page by score cursor
ZRANGE posts:{celeb:7} +inf 1727519000 BYSCORE REV LIMIT 0 20Interview problem
The problem
Instagram-style feed at scale
Design the home feed: 500M users, 100M daily active, average 200 followees, some accounts with 100M followers, feed loads under 200 ms. Discuss fan-out on write vs read, Redis's role, Kafka, the database, hot users and pagination.
The interviewer follows up
Twitter timeline: sorted set or list?
Explain it without notes
Explain the hybrid fan-out approach and why it's needed.
Practice
Estimate fan-out writes per second: 5K posts/sec, average 300 followers, 1% of posts from accounts with 5M followers.
Trade-offs
- ↔
Push: fast reads, heavy writes and memory. Pull: light writes, slower reads. Hybrid balances them with more complexity.
Done when you can
I can design a feed with hybrid fan-out, capped timelines and cursor pagination.