Command Palette

Search for a command to run...

Hectal
Phase 6Intermediate7 of 18 in Apache Kafka

Reliability Patterns

Retries with backoff and jitter, retry topics and dead-letter topics, poison messages, consumer lag and backpressure, ordering guarantees end to end, and hot partitions.

Real consumers fail: downstream services return 503, a malformed record can't be parsed, a traffic spike outruns processing. This phase is the toolbox for keeping a Kafka pipeline moving through all of that without losing records, reordering them unexpectedly, or melting the downstream.

It ends with the two hardest trade-offs in Kafka designs: ordering versus parallelism, and what to do when one key carries a third of all traffic.

0/5 · 0%
5 topics ~41 min 8 code blocks & diagrams
Start with the first topic
1
6.1

Consumer Retry Strategies: Backoff, Jitter, and Retry Storms

Transient failures (timeouts, 503s, deadlocks) deserve retries; permanent ones (bad data, validation errors) don't. Retry in place with exponential backoff and jitter for brief blips, move to delayed retry topics for longer outages, and never hammer a struggling downstream with immediate retries.

7 min 1 code practice

2
6.2

Retry Topics, Dead-Letter Topics, and Poison Messages

A poison message (one that always fails) must not block a partition forever. Route failing records to delayed retry topics (for example 5 s, 30 s, 5 min, 30 min), and after the last attempt to a dead-letter topic (DLT) with headers describing the original topic, partition, offset, exception and attempt count. Replay from the DLT after fixing the cause.

9 min 1 diagram 1 code practice

3
6.3

Consumer Lag and Backpressure

Lag is how far a group's committed offsets trail the end of the log. It grows whenever producers outpace consumers. Kafka handles this gracefully (records wait on disk), but lag means delay, and if it outgrows retention it means data loss. Diagnose why consumers are slow before adding consumers.

9 min 1 diagram 1 code practice

4
6.4

Ordering End to End: What Kafka Guarantees and What You Must

Kafka orders records within a partition. End-to-end order also needs the right key, one producer path per key with idempotence, consumers that process each partition (or key) sequentially, and retry strategies that don't reorder. Global ordering across everything requires a single partition, which caps throughput.

8 min 1 diagram practice

5
6.5

Hot Partitions and Skewed Keys

When one key (a huge customer, a viral product, a busy device) dominates traffic, its partition's broker and consumer are overloaded while others idle. Fix it by narrowing the key, bucketing hot keys with a suffix, splitting hot tenants onto dedicated capacity, or doing two-stage aggregation, each trading away some ordering or simplicity.

8 min 1 diagram 1 code practice