Where Exactly-Once Ends: Payments, Emails, and External Systems

In one line

Kafka transactions stop at Kafka's edge. A consumer that charges a card or sends an email can crash after the side effect but before committing its offset, and the retry repeats the side effect. Only idempotency at the external system (idempotency keys, deduplication records, natural idempotent operations) makes those effects happen once.

0/5 · 0%

Think of it like this

A waiter who takes an order to the kitchen and then faints before writing it on the order pad. When a colleague replays the pad, they send the order again: two identical meals. Only if the kitchen recognises "table 7, order 3 already cooked" is the duplicate avoided.

Key ideas

01
The gap: consume → call external service (charge, email, webhook) → commit offset. A crash between the call and the commit is unavoidable in distributed systems; the record will be redelivered.
02
Idempotency key: derive a stable key from the event (for example the eventId or paymentId) and pass it to the external service. Stripe-style payment APIs dedupe requests with the same key within a window; many email and messaging providers support similar keys.
03
Local dedup record: before the side effect, record intent; after, record completion (processed_events table with a unique event ID), in the same database transaction as any business update. Checking it on redelivery skips the repeat.
04
When the external system has no idempotency support: reduce the window (commit immediately after the call), accept rare duplicates and reconcile (compare provider records with your records), or put a system you control in front that dedupes.
05
Kafka transactions + external DB: you can't commit both atomically. Options: store offsets in the database in the same transaction as the business write (and seek to them on startup), or use the inbox pattern (next topic).

Code & diagrams

PaymentConsumer.javajava

@KafkaListener(topics = "payments.requested", groupId = "payment-executor")
public void onPaymentRequested(PaymentRequested evt, Acknowledgment ack) {
    // Stable key: the same event always maps to the same provider request.
    String idempotencyKey = "pay-" + evt.paymentId();
    ChargeResult result = provider.charge(evt.amount(), evt.cardToken(), idempotencyKey);
    // If we crash here and the event is redelivered, the provider returns the ORIGINAL result
    // for this idempotency key instead of charging again.
    payments.recordResult(evt.paymentId(), result);   // upsert, unique on paymentId
    ack.acknowledge();                                // commit offset last
}

eos-boundary.mermaiddiagram

Rendering diagram…

Interview problem

The problem

Charge once, email once

A consumer receives PaymentRequested, charges an external payment provider, then crashes before committing. The message is redelivered. How do you prevent a double charge? Separately, how do you prevent duplicate order-confirmation emails? Compare Kafka transactions, idempotency keys, an external database and the external API's own guarantees.

The interviewer follows up

Kafka says exactly-once; why can I still get duplicate emails?

When it breaks

Random UUID generated per attempt as the idempotency key

What you see

Each redelivery generates a new key, so the provider treats retries as new payments: double charges.

Fix & prevent

Derive the key deterministically from the event (paymentId or eventId).

Explain it without notes

Why can't Kafka transactions make a payment API call exactly-once?

Practice

Implement the payment consumer with a mock provider that dedupes by key, and kill the consumer after the charge but before the ack.

Trade-offs

↔
Dedup storage and idempotency plumbing cost effort, but they're the only real protection for external side effects.

Done when you can

I can explain the exactly-once boundary and make external side effects idempotent.

Command Palette