Topic 14.1
Reliability
In one line
Failover, redundancy, replication and health checks — the mechanisms that make downtime an exception, not a feature.
Think of it like this
A car built with a spare tyre, an airbag, and a warning light for low oil — it's designed assuming SOMETHING will eventually go wrong, and plans for that instead of pretending nothing ever will.
Key ideas
- 01
Failover: automatic role switch when a node fails (primary → replica promotion, LB → live nodes).
- 02
Redundancy: N+1 copies of critical components so no single failure is fatal (multi-AZ!).
- 03
Replication: copies of data (primary/standby) — the durability + availability half (Phase 8).
- 04
Health checks: the sensor network — L7 /healthz, readiness vs liveness, circuit breakers on dependencies.
- 05
The availability math: 99.9% = 8.7h/yr; 99.99% = 52min/yr; each 9 doubles the engineering bill roughly.
- 06
Reliability design = pre-mortem + chaos: assume every component fails; rehearse the failure (chaos engineering).
- 07
Interview: 'stateful tiers get replication + failover; stateless tiers get redundancy + LB health checks'.
Code & diagrams
The minute-by-minute picture of what actually happens when a primary dies at 2am.
Explain without notes
A DB primary dies at 2am. Walk the minutes: detection, failover, cutover, and what clients see.
Practice
Design the failover runbook for your app: LB health, DB promotion, cache reconnect, and the 10-minute budget.
Trade-offs
- ↔
Redundancy costs run-time money and adds coordination; MTTF pushes your SLA only when measured and rehearsed.
Run it in production
You've designed it. Now build, operate, and break the same idea hands-on in the DevOps courses:
Completion checklist
I can name failover/redundancy/replication/health as four distinct reliability levers.