System 12.32 — Web Crawler
In one line
Fetch billions of pages politely and efficiently: a URL frontier with per-host politeness, distributed fetchers, deduplication of URLs and content, and robust parsing, all without hammering any single website.
Think of it like this
A team of librarians collecting every newspaper in the world. They keep a to-visit list, never send two people to the same small shop at once (politeness), skip papers they already have (dedup), and follow every 'see also' reference they find (link extraction).
Key ideas
- 01
Requirements: ~1 B pages/month (~400 pages/s average), respect
robots.txtand crawl delays, prioritise important/fresh pages, detect duplicates, handle traps (infinite calendars, session-ID URLs), store raw content for indexing. - 02
URL FRONTIER: priority queues (by page importance/freshness) feeding per-HOST queues; a scheduler releases at most one request per host every N seconds. Each host queue is owned by one fetcher worker (hash host → worker), which also caches DNS and robots.txt per host.
- 03
DEDUP: a Bloom filter or large key-value set of normalised URLs ('seen?') before enqueueing (Phase 13B, Bloom filters); content fingerprints (hash, or SimHash for near-duplicates) to skip mirrors and boilerplate copies.
- 04
Pipeline: fetch → store raw HTML in object storage (S3) → parse (extract text and links) → normalise links → filter (robots, domain rules, trap detection like max depth/URL length) → back into the frontier; parsed text goes to the indexer via Kafka. Recrawl frequency per page is based on how often it changed before.
Code & diagrams
Explain without notes
How do you enforce politeness across thousands of fetcher threads?
Practice
How would you avoid crawler traps such as infinite calendar pages?
Trade-offs
- ↔
Breadth vs freshness: crawling new pages vs recrawling changed ones competes for the same budget; prioritise by importance and change rate.
Run it in production
You've designed it. Now build, operate, and break the same idea hands-on in the DevOps courses:
Completion checklist
I can explain the frontier, politeness, and dedup layers