Command Palette

Search for a command to run...

Hectal
Phase 3Intermediate4 of 18 in Apache Kafka

Consumers & Consumer Groups

The poll loop, offsets and commits, consumer groups and scaling limits, rebalancing protocols (eager, cooperative, static membership, the new KIP-848 protocol), and processing models that don't break ordering.

Consumers decide delivery semantics. When you commit an offset relative to processing decides whether a crash loses records or duplicates them, and how you thread your processing decides whether per-key ordering survives.

This phase covers the consumer mechanics precisely, then the group protocol that assigns partitions, because rebalances are behind a large share of real Kafka incidents.

0/5 · 0%
5 topics ~41 min 8 code blocks & diagrams
Start with the first topic
1
3.1

The Consumer and the Poll Loop

A consumer subscribes to topics, calls poll() in a loop to fetch batches of records from the partitions it's assigned, deserializes them, processes them, and commits offsets. poll() also drives group membership, so the loop's timing matters as much as its logic.

7 min 1 code practice

2
3.2

Offsets, Commits, and Lag

A committed offset is the position of the next record the group should read, stored in __consumer_offsets. Commit after processing for at-least-once; commit before for at-most-once. Committing an offset means "everything before this is done", so you can't skip a failed record and commit past it without deciding what happens to it.

9 min 2 code practice

3
3.3

Consumer Groups and Scaling

Within a consumer group, each partition is assigned to exactly one consumer, so a group's parallelism is capped by the partition count; extra consumers sit idle. Different groups read the same topic independently, each with its own offsets.

8 min 1 diagram 1 code practice

4
3.4

Rebalancing: Protocols, Timeouts, and Storms

A rebalance reassigns partitions when members join, leave, crash or exceed the poll interval. Eager rebalancing stops the whole group; cooperative rebalancing moves only affected partitions; static membership avoids rebalances on restarts; and Kafka 4.0's new consumer protocol (KIP-848) moves assignment to the broker with incremental, per-member reconciliation.

9 min 1 diagram 1 code practice

5
3.5

Processing Models: Threads, Pause/Resume, Long Work

You can process records single-threaded per consumer, with one consumer per thread, or with a worker pool behind one polling thread. Parallelism inside a partition must be per key to keep ordering, offsets must be committed only when contiguous work is done, and long processing needs pause/resume so polling continues.

8 min 1 code practice