Rack Awareness and Surviving an AZ Failure

In one line

Setting broker.rack to each broker's availability zone makes Kafka spread each partition's replicas across zones, so one zone outage never takes all copies. With RF=3 across 3 AZs and min.insync=2, the cluster keeps accepting durable writes through a full AZ loss. Follower fetching lets consumers read from their own AZ to cut cross-AZ costs.

0/4 · 0%

Think of it like this

A company keeping copies of critical files in three separate buildings. A fire in one building still leaves two copies, and staff in each building read their local copy instead of driving across town.

Key ideas

01
broker.rack=us-east-1a (per broker) makes replica assignment place replicas of each partition in different racks when possible. Without it, two replicas can land in the same AZ.
02
Three AZs, RF=3, min.insync=2, acks=all: an AZ outage removes one replica of each partition; ISR = 2 ≥ min.insync, so writes continue and nothing acknowledged is lost. Controllers must also be spread across the three AZs so the KRaft quorum keeps a majority.
03
Two AZs are awkward: RF=3 puts two replicas in one AZ; losing that AZ leaves one replica, below min.insync=2, so writes stop; and the controller quorum can't keep a majority in a 2-zone split. Use three zones (or a stretch design with a third-zone tiebreaker for controllers).
04
Follower fetching (KIP-392): set replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector on brokers and client.rack on consumers so consumers read from a replica in their own AZ. Cross-AZ data transfer is billed in most clouds, so this can save a lot.
05
Cost: producer-to-leader and leader-to-follower traffic still crosses AZs; RF=3 means roughly 2× the produced bytes cross zones for replication.

Code & diagrams

three-az.mermaiddiagram

Rendering diagram…

rack.propertiesproperties

# broker (per AZ)
broker.rack=ap-south-1a
replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector
default.replication.factor=3
min.insync.replicas=2
# consumer
client.rack=ap-south-1a

Interview problem

The problem

Survive one AZ failure

Design Kafka to survive the loss of one availability zone: choose RF, replica placement, acks and min.insync.replicas, and explain each decision. Include controllers and consumers.

The interviewer follows up

Why not RF=2 across two AZs to save money?

Explain it without notes

What does broker.rack change, and what does follower fetching save?

Practice

Create a topic in the lab with each broker in a different broker.rack and verify replicas of every partition are in different racks.

Trade-offs

↔
Multi-AZ placement survives zone failures but pays cross-AZ network costs for replication.

Done when you can

I can design an AZ-failure-tolerant Kafka deployment including controllers and consumers.

Command Palette