Command Palette

Search for a command to run...

Hectal
Phase 5Advanced6 of 18 in Apache Kafka

Delivery Semantics & Transactions

At-most-once, at-least-once and exactly-once precisely; Kafka transactions and read-process-write; where exactly-once ends (external APIs, databases, emails); and the inbox pattern for idempotent consumers.

"Exactly-once" is the most misunderstood phrase in Kafka. Kafka offers exactly-once for reading from Kafka, processing, and writing back to Kafka. It can't make an email provider, a payment gateway or your database exactly-once by itself.

This phase makes the guarantees precise and gives you the patterns (idempotency keys, inbox tables, transactional writes) that close the gap for real side effects.

0/5 · 0%
5 topics ~39 min 8 code blocks & diagrams
Start with the first topic
1
5.1

At-Most-Once, At-Least-Once, Exactly-Once

The order of "process" and "commit offset" decides delivery semantics. Commit then process: a crash loses records (at-most-once). Process then commit: a crash reprocesses records (at-least-once, the usual default). Exactly-once needs either transactions (Kafka to Kafka) or idempotent processing that makes duplicates harmless.

7 min 1 diagram practice

2
5.2

Kafka Transactions

A transactional producer (with a stable transactional.id) can write to several partitions and commit consumer offsets as one atomic unit: either all records and the offset commit become visible, or none do. Consumers with isolation.level=read_committed skip aborted records. A transaction coordinator on the brokers manages the protocol.

8 min 1 diagram 1 code practice

3
5.3

Exactly-Once Stream Processing (Read-Process-Write)

For pipelines that read from Kafka and write back to Kafka, exactly-once is a solved problem: wrap output writes and input offset commits in one transaction, or use Kafka Streams with processing.guarantee=exactly_once_v2, which also covers state store changelogs.

7 min 1 code practice

4
5.4

Where Exactly-Once Ends: Payments, Emails, and External Systems

Kafka transactions stop at Kafka's edge. A consumer that charges a card or sends an email can crash after the side effect but before committing its offset, and the retry repeats the side effect. Only idempotency at the external system (idempotency keys, deduplication records, natural idempotent operations) makes those effects happen once.

9 min 1 diagram 1 code practice

5
5.5

The Inbox Pattern and Idempotent Consumers

The transactional inbox makes a consumer's database updates idempotent: in one database transaction, insert the event ID into an inbox (processed-events) table with a unique constraint and apply the business change. A redelivered event fails the unique insert and is skipped, so the effect happens once even with at-least-once delivery.

8 min 2 code practice