Idempotency: Keys, Deduplication and Retry-Safe Writes

In one line

Networks time out, clients retry and consumers redeliver, so every write that matters must be safe to apply twice. Idempotency keys stored with a unique constraint make a repeated request return the original result; deduplication tables make consumers ignore redelivered messages; natural unique constraints and conditional updates make many operations idempotent by design.

0/5 · 0%

Think of it like this

A lift button. Pressing it five times calls the lift once. A payment button should behave the same way, but by default a double-click can charge twice.

Key ideas

01
HTTP semantics: GET, PUT and DELETE are idempotent by definition; POST is not. Make POSTs idempotent with an Idempotency-Key header (Stripe-style): the server stores the key with the request hash and the response.
02
Implementation: idempotency_key(key PK, request_hash, status, response, created_at). On a request, insert the key (the unique constraint serialises duplicates); if it exists and has completed, return the stored response; if in progress, return 409 or wait; if the request hash differs, reject (the key was reused for a different request).
03
Do the business write and the key's completion in the same transaction, so a crash can't leave the key completed without the effect, or vice versa. Expire keys after a retention window (e.g. 24 h to 7 days).
04
Consumers: a processed_message(message_id PK) table written in the same transaction as the effect; or natural idempotency (INSERT ... ON CONFLICT DO NOTHING on a business key, UPDATE ... WHERE version = ?, state transitions that only apply from the expected state).
05
Webhooks: providers retry; store the provider's event ID with a unique constraint before acting. Notifications: dedup by (user, template, entity, time bucket) to avoid duplicate sends.

Code & diagrams

idempotency.sqlsql

CREATE TABLE idempotency_key (
  key          text PRIMARY KEY,
  request_hash text NOT NULL,
  status       text NOT NULL CHECK (status IN ('in_progress','completed')),
  response     jsonb,
  created_at   timestamptz NOT NULL DEFAULT now()
);

-- step 1: claim the key (a duplicate request gets 0 rows)
INSERT INTO idempotency_key (key, request_hash, status)
VALUES ('pay_7f3c9a', 'sha256:ab12...', 'in_progress')
ON CONFLICT (key) DO NOTHING
RETURNING key;

-- step 2 (same transaction as the payment record)
BEGIN;
INSERT INTO payment (id, order_id, amount, status) VALUES (5501, 9001, 1499.00, 'authorised');
UPDATE idempotency_key
   SET status = 'completed', response = '{"paymentId": 5501, "status": "authorised"}'
 WHERE key = 'pay_7f3c9a';
COMMIT;

-- state transition that is naturally idempotent
UPDATE orders SET status = 'paid' WHERE id = 9001 AND status = 'placed';   -- 0 rows on retry

Interview problem

The problem

Idempotent payment API

Clients call POST /payments and sometimes retry after a timeout, causing double charges. The server calls an external card processor. Design an end-to-end idempotent flow.

When it breaks

Dedup check and effect in separate transactions

What you see

A crash after the effect but before recording the message ID leads to reprocessing and a duplicate side effect; the reverse order loses the message.

Fix & prevent

Record the key or message ID and apply the effect in the same database transaction; for external effects, pass the key to the external system.

Explain it without notes

Why does the idempotency record need the request hash?

Practice

Make a Kafka consumer that sends welcome emails idempotent.

Trade-offs

↔
Idempotency storage and checks add writes and latency; without them, retries cause duplicate money movements and messages.

Done when you can

I can design idempotent APIs, consumers and webhooks with correct transactional boundaries.

Command Palette