Replication, the ISR, and the High Watermark

In one line

Each partition has a leader and followers that fetch from it. Followers caught up within replica.lag.time.max.ms form the in-sync replica set (ISR). The high watermark is the highest offset replicated to all ISR members; consumers only see records below it, so they never read data that could disappear in a failover.

0/4 · 0%

Think of it like this

A teacher dictating notes to three students. The notes count as "official" only up to the last line all attentive students have written down. A student who daydreams too long is marked as not keeping up until they catch up.

Key ideas

01
Followers replicate by sending fetch requests to the leader, just like consumers. The leader tracks each follower's fetch position.
02
ISR membership: a follower is in sync if it has caught up to the leader's log end within replica.lag.time.max.ms (30 s). Slow disks, network problems, GC pauses or a crashed broker make it fall out (ISR shrink); when it catches up, it rejoins (ISR expand).
03
High watermark (HW): the minimum log end offset across the ISR. Records above the HW exist on the leader but aren't committed; consumers can't read them (they'd risk reading something that disappears if the leader fails). With acks=all, the producer is acknowledged when the HW passes its records.
04
Leader epochs: each leadership change increments an epoch, and followers use epochs to truncate divergent log suffixes after failover, replacing older high-watermark-based truncation, which could lose or diverge data.
05
Monitor: UnderReplicatedPartitions (ISR smaller than the replica set), UnderMinIsrPartitionCount (writes with acks=all will fail), ISR shrink/expand rates, and follower fetch lag.

Code & diagrams

hw.txttext

Leader B1 log:     0 1 2 3 4 5 6 7 8     LEO=9
Follower B2 log:   0 1 2 3 4 5 6 7       LEO=8
Follower B3 log:   0 1 2 3 4 5           LEO=6   (still within replica.lag.time.max.ms -> in ISR)

ISR = {B1, B2, B3}
High watermark = min(9, 8, 6) = 6  -> consumers can read offsets 0..5
acks=all producers of offsets 6..8 are still waiting for their acknowledgement

urp.shbash

kafka-topics.sh --bootstrap-server $B --describe --under-replicated-partitions
  Topic: payments  Partition: 4  Leader: 2  Replicas: 2,1,3  Isr: 2,3
  Topic: payments  Partition: 9  Leader: 3  Replicas: 3,1,2  Isr: 3,2

kafka-topics.sh --bootstrap-server $B --describe --under-min-isr-partitions
# empty: every partition still has at least min.insync.replicas in sync

Explain it without notes

Why can't consumers read records above the high watermark?

Practice

In the lab, pause one broker container (docker pause kafka3) for 40 s and watch the ISR shrink and expand.

Trade-offs

↔
A longer replica.lag.time.max.ms tolerates brief follower hiccups but lets slower followers stay in the ISR, which can slow acks=all writes.

Done when you can

I can explain leaders, followers, ISR, high watermark and leader epochs.