The Inbox Pattern and Idempotent Consumers

In one line

The transactional inbox makes a consumer's database updates idempotent: in one database transaction, insert the event ID into an inbox (processed-events) table with a unique constraint and apply the business change. A redelivered event fails the unique insert and is skipped, so the effect happens once even with at-least-once delivery.

0/5 · 0%

Think of it like this

A receiving desk stamping every delivery note's number into a logbook as it books the goods into stock, both in one motion. When the same delivery note shows up again, the logbook says "already booked", so stock isn't counted twice.

Key ideas

01
Table: inbox(event_id PRIMARY KEY, topic, partition, offset, processed_at). Handler: BEGIN; INSERT INTO inbox ...; UPDATE business tables ...; COMMIT; then commit the Kafka offset. On a duplicate event, the insert violates the key and the handler skips the business update.
02
Why it works: the dedup record and the business effect are atomic in the same database. A crash before commit leaves neither; after commit, both exist. The Kafka offset commit can then be at-least-once safely.
03
Natural idempotency is even simpler when available: UPSERT of the latest state, UPDATE ... SET status='SHIPPED' WHERE status <> 'SHIPPED', or version checks (WHERE version < :v). Increments (balance = balance + x) are not idempotent without an inbox.
04
Cleanup: delete inbox rows older than the maximum redelivery window (retention plus replay horizon), or partition the table by date.
05
Pairs with the outbox: outbox makes publishing reliable on the producer side; inbox makes consumption idempotent on the consumer side.

Code & diagrams

inbox.sqlsql

CREATE TABLE inbox (
  event_id     uuid PRIMARY KEY,
  topic        text NOT NULL,
  partition    int  NOT NULL,
  "offset"     bigint NOT NULL,
  processed_at timestamptz NOT NULL DEFAULT now()
);

BEGIN;
INSERT INTO inbox(event_id, topic, partition, "offset")
VALUES ('7f1c9e2a-5b6d-4a31-9c1e-2d8f7b3a6e10', 'inventory', 3, 88121)
ON CONFLICT (event_id) DO NOTHING;
-- If the insert affected 0 rows: duplicate, ROLLBACK and skip.
UPDATE stock SET reserved = reserved + 2 WHERE sku = 'sku-7';
COMMIT;

InboxHandler.javajava

@Transactional
public void handle(ConsumerRecord<String, InventoryEvent> r) {
    int inserted = jdbc.update(
        "INSERT INTO inbox(event_id, topic, partition, \"offset\") VALUES (?,?,?,?) ON CONFLICT DO NOTHING",
        r.value().eventId(), r.topic(), r.partition(), r.offset());
    if (inserted == 0) {
        log.info("duplicate event {} skipped", r.value().eventId());
        return;
    }
    stock.reserve(r.value().sku(), r.value().qty());   // same DB transaction
}

Interview problem

The problem

Inventory reservations counted twice

An inventory consumer increments reserved on each OrderCreated. After a rebalance, some orders were reserved twice. Fix it so reservations are applied exactly once, and explain how cleanup and replays work.

Explain it without notes

Why must the inbox insert and the business update be in the same transaction?

Practice

Add an inbox to a consumer and replay its topic from the beginning with a new offset reset; verify no double effects.

Trade-offs

↔
The inbox costs a write per event and table maintenance; natural idempotency is cheaper where the data model allows it.

Done when you can

I can implement the inbox pattern and prefer natural idempotency where possible.

Command Palette