System 12.21 — Distributed Logging
In one line
Collecting logs from 10k services into one searchable sink — the ELK/opentelemetry story.
Think of it like this
A company-wide CCTV system for software. Every service constantly writes down what it's doing; when something breaks, engineers search through this recorded footage to figure out what happened and when.
Key ideas
- 01
Agents (sidecars/daemons) tail logs → local buffer → centralized pipeline (Kafka) → indexer (ES/ClickHouse) → UI.
- 02
Critical decisions: sampling (why log everything when 99% is noise?), retention tiers (hot/warm/cold), structured logs (JSON!).
- 03
Correlation: traceId injected per request (X-Request-Id) — the single line that makes 10k-service debugging possible.
- 04
Pipeline: agents → Kafka topics per level → consumers parse/normalize → ES batched writes; backpressure via partitions.
- 05
Scale: 1TB/day → ~12 MB/s sustained; burst 50x on incidents: queue is the shock absorber.
- 06
Failures: ES slow under heavy writes → buffer or drop (sampling back); Kafka retention holds 30d for replay.
- 07
Interviews: the interesting part is 'how do you lose LESS log' — agents, batching, structured fields, and queue buffering.
Code & diagrams
Every service writes locally first — nothing waits on the network just to write a log line.
Explain without notes
Why must structured (JSON) logs come before the 10kth service, not after — what costs you pay to retrofit?
Practice
Design the JSON log schema (ts, level, service, traceId, spanId, fields) and the sampling policy.
Trade-offs
- ↔
Index everything = money; sample aggressively = missing evidence during incidents. Tiered retention is the compromise.
Run it in production
You've designed it. Now build, operate, and break the same idea hands-on in the DevOps courses:
Completion checklist
I can present agent→buffer→index→query logging with traceId correlation.