System 12.37 — Ad Click Aggregation & Top-K Heavy Hitters
In one line
Count billions of ad clicks per day accurately enough to bill advertisers and show live dashboards: stream aggregation with windows and watermarks, exactly-once-ish processing with reconciliation, and top-K ads per minute.
Think of it like this
Counting votes on election night. Early counts arrive live on TV (streaming, approximate, fast), while the official count is recounted carefully afterwards (batch reconciliation, exact). Money depends on the official count.
Key ideas
- 01
Requirements: 1 B clicks/day (~10 k/s, peaks 50 k/s), per-ad click counts per minute, top 100 ads in the last minute/hour, filters (country, device), results within ~1 minute, correctness for billing, dedup of fraudulent/duplicate clicks.
- 02
Pipeline: click events → Kafka (partitioned by ad ID) → stream processor (Flink) with 1-minute TUMBLING WINDOWS and WATERMARKS for late events → aggregated counts to an OLAP/time-series store (ClickHouse, Druid, Pinot) for dashboards. Top-K per window: a min-heap per partition merged globally, or a Count-Min Sketch + heap for huge cardinalities (Phase 13B, probabilistic structures).
- 03
CORRECTNESS: exactly-once state in the stream processor (checkpoints + transactional sinks) plus deduplication by click ID; raw events also land in the data lake, where a daily BATCH job recomputes exact counts and RECONCILES with the streaming results before invoicing (Phase 13B, batch vs stream).
- 04
Hot ads: a viral ad can overload one partition; pre-aggregate with a random sub-key (
adId#0..9) and merge in a second stage.
Code & diagrams
Explain without notes
Why do you still need a batch job if the stream is exactly-once?
Practice
How do you handle a click that arrives 20 minutes late?
Trade-offs
- ↔
Freshness vs exactness: streaming gives minute-level answers with small errors; batch gives exact numbers hours later. Billing systems use both.
Run it in production
You've designed it. Now build, operate, and break the same idea hands-on in the DevOps courses:
Completion checklist
I can design windows, watermarks, dedup, top-K, and reconciliation