The Transactional Outbox: Ending Dual Writes

In one line

Writing to the database and publishing to Kafka as two separate steps can leave them inconsistent. The outbox pattern writes the business change and an event row in one database transaction; a relay (polling or CDC) publishes outbox rows to Kafka. Events are published if and only if the transaction committed.

0/4 · 0%

Think of it like this

A shop that writes each sale in the ledger and on a "to announce" list in the same stroke. A runner later reads the list and makes the announcements. If the shop burns down mid-sale, there's either no ledger entry and no announcement, or both.

Key ideas

01
The dual-write failures: DB commit succeeds, Kafka publish fails → downstream never learns of the order. Kafka publish succeeds, DB transaction rolls back → downstream acts on an order that doesn't exist. Publishing inside the DB transaction doesn't help: Kafka and the database can't commit atomically together.
02
Outbox table: outbox(id, aggregate_type, aggregate_id, type, payload, created_at). In the same transaction as the business write, insert the event. Commit once.
03
Relay options: CDC with Debezium's outbox event router SMT (reads outbox inserts from the WAL, routes by aggregate type, uses aggregate ID as key); or a polling publisher (SELECT ... FOR UPDATE SKIP LOCKED, publish, mark sent). CDC gives lower latency and no polling load.
04
Guarantees: at-least-once publishing (a relay crash after publish but before marking may republish), so events carry IDs and consumers are idempotent (inbox). Ordering per aggregate is preserved by keying on aggregate ID and publishing in insert order.
05
Housekeeping: delete published outbox rows (or with Debezium, delete immediately after insert in the same transaction, since the WAL still has them).

Code & diagrams

outbox-flow.mermaiddiagram

Rendering diagram…

OrderService.javajava

@Transactional
public Order placeOrder(PlaceOrder cmd) {
    Order order = orders.save(Order.from(cmd));
    outbox.save(new OutboxEvent(
        UUID.randomUUID(),                 // eventId for consumer dedup
        "order", order.id(),              // aggregate type and id -> topic and key
        "OrderCreated",
        json.write(OrderCreated.from(order))));
    return order;                          // one commit: order + event, or neither
}

outbox-router.propertiesproperties

transforms=outbox
transforms.outbox.type=io.debezium.transforms.outbox.EventRouter
transforms.outbox.table.field.event.key=aggregate_id
transforms.outbox.route.by.field=aggregate_type
transforms.outbox.route.topic.replacement=commerce.${routedByValue}s
table.include.list=public.outbox

Interview problem

The problem

Save the order and publish OrderCreated consistently

The order service must save an order to PostgreSQL and publish OrderCreated to Kafka. How do you avoid inconsistent dual writes? Compare direct dual write, the transactional outbox, and CDC.

When it breaks

Publishing to Kafka inside the @Transactional method before commit

What you see

If the DB commit later fails (constraint violation, deadlock), consumers have already received an event for an order that doesn't exist.

Fix & prevent

Outbox, or at minimum publish after commit with a durable retry, accepting the loss window.

Explain it without notes

Why can't you make a database write and a Kafka publish atomic directly?

Practice

Implement the outbox with a polling relay, kill the relay after publishing but before marking rows sent, and verify consumers dedupe the republished events.

Trade-offs

↔
The outbox adds a table, a relay and at-least-once semantics, and removes dual-write inconsistency entirely.

Done when you can

I can explain dual-write failures and implement the outbox with a CDC or polling relay.

Command Palette