Designing for Failure: The Five Things That WILL Break
Your HLD will break in five predictable ways. Here's the pre-mortem list — design the failure handling before the interviewer asks.
Senior design answers aren't the ones where nothing breaks — those don't exist. They're the ones where the candidate has already broken everything in their head, twice, and came back with a fix each time.
1. A node dies
Service instances die daily. So: stateless services so any instance can serve any request, health checks + auto-replacement, and state pushed to durable stores (DB, Redis with AOF/replicas). Never store something critical in a local variable and call it done.
2. A call hangs
The default Java timeout for an unconfigured HTTP client is effectively 'forever'. Design everyone's call with explicit timeouts + retries-with-backoff, and when the downstream dies, a circuit breaker trips so the failing dependency stops being called at all (Resilience4j, Sentinel, or a manual half-open state machine).
3. A message duplicates
Kafka gives at-least-once under retries — meaning duplicates are the DEFAULT contract. The production retort is idempotency on the consumer: upsert by natural key, or store processed event ids and skip. Payment systems scream this louder than anything else.
4. Traffic spikes 20x
Auto-scaling reacts in minutes, but the spike is in seconds. The pre-mortem answer: capacity headroom + per-service rate limits + a queue to absorb the burst + degrade features (disable recommendations, serve cached inventory) instead of serving errors.
5. A deployment makes things worse
Canaries before full rollouts, feature flags to turn things off without a deploy, and rollback as a first-class button. The '5 whys' of the worst production outage is usually that deploying new code and fixing old code fought each other.
The pre-mortem is the cheapest reliability work you'll ever do. Break everything on paper now so nothing breaks at 3 a.m. for real.