System 12.24 — Distributed Job Scheduler
In one line
Cron at fleet scale: at-least-once scheduling, leases, leader election for periodic jobs, and exactly-one-owner semantics.
Think of it like this
The office's central calendar/reminder system, but for computers: 'run this backup every night at 2 AM' or 'send this reminder in exactly one hour,' reliably, even if one server crashes at the wrong moment.
Key ideas
- 01
Core: a job registry (name, cron/interval, payload, timeout) + a scheduler picking due jobs + workers executing.
- 02
Leadership: exactly one scheduler node owns the trigger loop per job (leader election — etcd/ZK consensus, Phase 13.7).
- 03
Execution lease: a job instance carries a lease (heartbeat); if the worker dies, the job is re-executed (at-least-once → idempotent handlers).
- 04
Catch-up & misfire policy: when the scheduler was down, fire_all vs single-fire vs skip — the config must be explicit.
- 05
Distributed lock alternative: for 'must run once across N workers', lease + fencing token (11.16).
- 06
Queue depth: jobs queue per trigger (Kafka/RQ); backpressure and priorities; retry + DLQ per job-type policy.
- 07
The honest guarantee sentence: 'exactly-once is a lie — I do at-least-once with idempotent handlers and leases'.
- 08
Extras: cron expression parsing, job history table, backfill tooling, per-tenant rate limits.
Code & diagrams
A leader decides WHAT runs when; a pool of workers actually runs it — separating the two is the whole design.
Explain without notes
The scheduler was down for 90 seconds and three jobs were due. Walk the missed-trigger policy decision.
Practice
Design the job table + lease heartbeat + retry policy for a nightly report pipeline.
Trade-offs
- ↔
Firing twice (safe, idempotent) vs never firing (silent miss) — the lock/lease balance is this system's soul.
Run it in production
You've designed it. Now build, operate, and break the same idea hands-on in the DevOps courses:
Completion checklist
I can present a scheduler with leadership, leases, misfire rules, and honest delivery semantics.