System 12.20 — Search Engine
In one line
The one HLD that's genuinely different: crawling, indexing, ranking, and serving billions of queries.
Think of it like this
How Google finds the right pages out of billions in under a second: it doesn't search live, it keeps a pre-built giant index (like the back of a book) ready in advance.
Key ideas
- 01
Pipeline: crawl (discover links) → parse/clean → index (inverted index: term → doc list) → query serving (rank on request).
- 02
Inverted index: word → posting list (docId, tf, positions); the core data structure — everything else is rank math.
- 03
Ranking: TF-IDF (term frequency × inverse doc frequency) → BM25 → query-stage + ML rerank; freshness boost.
- 04
Serving layer: shard index across nodes, replicate hot shards, cache top results (the 'top 1k' cache), query aggregation.
- 05
Crawl budget: politeness (robots.txt), refresh scheduling by importance (dynamic pages re-crawled hourly).
- 06
Scale: 8B pages indexed, 100k qps peak — index shards + replica rings + massive caching of popular queries.
- 07
Query flow: parse → expand (synonyms) → near/terms → shard fan-out (or a replicated index per node) → merge + rank → highlight + cache.
- 08
Interview shortcut: present the inverted index, then partitioning fan-out + caching, then spell-out the crawler cadence.
Java / Spring map
- →
Lucene/Elastic are textbook implementations: analyze, inverted index, BM25 — say 'I'd use ES/Lucene semantics'.
Code & diagrams
The trick that makes search fast: all the expensive work (crawling, indexing) happens BEFORE you ever type a query.
Explain without notes
What does the inverted index give you that a LIKE query never can, in latency terms?
Practice
Design the index sharding + the query-fan-out merge, plus the popular-query cache.
Trade-offs
- ↔
Index freshness vs crawl cost; ranking sophistication vs serving latency — the whole field is dials.
Run it in production
You've designed it. Now build, operate, and break the same idea hands-on in the DevOps courses:
Completion checklist
I can present crawl→index→serve with the inverted index at the center.