Topic 3.6
Share Groups: Queue-Style Consumption (KIP-932)
In one line
Share groups (KIP-932) let many consumers read the same partition together, acknowledging records one by one, like a work queue: records are locked while a consumer works on them, redelivered if not acknowledged in time, and given up after a delivery limit. They lift the "one consumer per partition" ceiling for work that doesn't need ordering. They arrived as early access in Kafka 4.0 and preview in 4.1, so check your version before relying on them.
Think of it like this
A ticket counter where any free clerk takes the next customer, instead of each clerk owning a fixed lane. If a clerk walks away mid-service, the customer goes back to the front of the line for the next clerk.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Share group
- A Kafka group where consumers share partitions and acknowledge records individually.
- KIP-932
- The Kafka Improvement Proposal that added queue-style share groups.
- Acquisition lock
- A time-limited claim on fetched records by one share consumer.
- ACCEPT / RELEASE / REJECT
- The three ways a share consumer can acknowledge a record.
- Delivery count
- How many times a record has been handed out; capped by a limit.
- KafkaShareConsumer
- The Java client class for consuming as part of a share group.
Step by step
01Why the restaurant notifier wants a queue
Restaurant notifications are independent jobs: order doesn't matter between restaurants, and each tablet call is slow. With a classic group, ordering.orders has 24 partitions, so at most 24 consumers work at once. A share group lets 60 workers pull from the same partitions during lunch.
02A share consumer
The code looks like a normal poll loop, but each record is acknowledged individually. A slow tablet API leads to RELEASE (try again later); a restaurant that no longer exists leads to REJECT.
Preview API: names and configs follow KIP-932 as implemented in 4.1 and may change. Enabling share groups on brokers is version-specific; follow your release notes.
Properties p = new Properties();
p.put("bootstrap.servers", "kafka1:9092,kafka2:9092,kafka3:9092");
p.put("group.id", "restaurant-notifier-share");
p.put("key.deserializer", StringDeserializer.class.getName());
p.put("value.deserializer", StringDeserializer.class.getName());
p.put("share.acknowledgement.mode", "explicit"); // acknowledge each record ourselves
try (KafkaShareConsumer<String, String> consumer = new KafkaShareConsumer<>(p)) {
consumer.subscribe(List.of("ordering.orders"));
while (running) {
for (ConsumerRecord<String, String> r : consumer.poll(Duration.ofMillis(500))) {
try {
tablets.notify(parse(r.value())); // idempotent by orderId
consumer.acknowledge(r, AcknowledgeType.ACCEPT);
} catch (TabletTimeoutException e) {
consumer.acknowledge(r, AcknowledgeType.RELEASE); // another worker may retry it
} catch (UnknownRestaurantException e) {
consumer.acknowledge(r, AcknowledgeType.REJECT); // never deliver again
}
}
consumer.commitSync(); // send the acknowledgements
}
}03Inspecting a share group
Share groups have their own admin tool, showing members and the start offset of records still in flight per partition.
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Using a share group for ordered events
The delivery service, which applies Created → Paid → RiderAssigned in order, is switched to a share group to get more parallelism.
Myth vs fact
Myth
Share groups turn Kafka into RabbitMQ.
Fact
They add queue-style consumption to Kafka's log: records still live in partitions with retention and can be replayed. Features like priorities and per-message routing still aren't there.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Keep handlers idempotent with share groups too. Lock expiry and RELEASE mean a record can be delivered to several consumers over time, so duplicates are part of the model.
Remember this
- 1
Consumer groups vs share groups: in a consumer group each partition has one owner and ordering per partition is kept; offsets are a single position. In a share group several consumers can fetch from the same partition, records are handed out individually, and each record is acknowledged on its own.
- 2
Acquisition locks: when a share consumer fetches records, they're locked to it for
group.share.record.lock.duration.ms(30 s by default). If it doesn't acknowledge in time, the lock expires and the records become available to others. - 3
Acknowledgement types:
ACCEPT(done),RELEASE(give it back for another try),REJECT(can't be processed; don't deliver again). Each delivery increments a delivery count; aftergroup.share.delivery.count.limit(5 by default) the record is archived instead of redelivered. - 4
What you give up: no ordering guarantee, even within a partition, and no simple single offset per partition. Use share groups for independent jobs (sending notifications, resizing images), not for state machines that need per-key order.
- 5
Status: early access in Kafka 4.0 and preview in 4.1; the API (
KafkaShareConsumer) and configs may still change, and some features (like a built-in DLQ for rejected records) are still being worked on. Read the release notes for your version before production use.
Explain it without notes
How do share groups differ from consumer groups?
What happens to a record a share consumer never acknowledges?
Practice
On a Kafka 4.1 lab with share groups enabled, run three share consumers on a 1-partition topic and show all three receive records.
Decide which of Tiffin's consumers could move to share groups and which must not.
Trade-offs
- ↔
Share groups give queue-like scaling and per-record acknowledgement without changing topics, at the cost of ordering, a newer and still-evolving API, and more broker-side state per record.
Done when you can
I can explain share groups, acquisition locks and acknowledgement types.
I only use share groups for work that doesn't need ordering.
I check the share-group status for my Kafka version before production use.