Redis Next to PostgreSQL and Kafka

In one line

In most architectures PostgreSQL is the source of truth, Kafka is the durable event backbone, and Redis holds fast derived state. The hard part is keeping them consistent: write to the database first, publish changes through an outbox or CDC, and let consumers update Redis idempotently.

0/6 · 0%

Think of it like this

A bank branch. The ledger in the vault is the truth (PostgreSQL), the internal mail system carries every transaction slip to other departments in order and keeps copies (Kafka), and the teller's quick-reference card of today's balances (Redis) is updated from those slips. If the card is lost, you rebuild it from the ledger.

Key ideas

01
Dual writes are the classic bug: writing to the database and then to Kafka (or Redis) in application code can fail halfway (crash, timeout), leaving systems inconsistent with no record of what was missed.
02
Transactional outbox: in the same database transaction as the business change, insert an event row into an outbox table. A relay (or Debezium CDC) publishes outbox rows to Kafka and marks them sent. Consumers update Redis (caches, counters, leaderboards, search indexes).
03
CDC directly from tables: Debezium reads the WAL and publishes row changes to Kafka; a consumer invalidates or updates Redis keys. Catches every writer, including batch jobs.
04
Idempotent consumers: Kafka delivers at least once, so Redis updates must be idempotent: set absolute values rather than increment, or record processed event IDs (SET processed:{eventId} 1 NX EX 86400), or use versioned writes (Topic 6.3).
05
Rebuildability: because Redis state is derived, you can rebuild it by replaying Kafka (within retention) or re-reading the database. Design for that: make rebuild jobs part of the system, not an afterthought.

Code & diagrams

outbox.mermaiddiagram

Rendering diagram…

outbox.sqlsql

BEGIN;
UPDATE orders SET status = 'PAID', version = version + 1 WHERE id = 9;
INSERT INTO outbox (id, aggregate_type, aggregate_id, type, payload)
VALUES (gen_random_uuid(), 'order', 9, 'OrderPaid',
        '{"orderId": 9, "version": 7, "amount": 1299}');
COMMIT;

Interview problem

The problem

Order service: what goes where

The order service must: store orders reliably, show a user's recent orders fast, update a live "orders per minute" dashboard, notify email and analytics services, and keep an inventory reservation hold. Decide what lives in PostgreSQL, Kafka and Redis, and how they stay in sync.

The interviewer follows up

Why not have the API write to Redis directly after the DB commit?

When it breaks

Dual write: DB commit succeeds, the Kafka publish times out

What you see

The order exists but no event was sent: no email, analytics missing, Redis dashboard low, and nothing records the gap.

Fix & prevent

Transactional outbox or CDC so events are derived from committed data.

Explain it without notes

Explain the transactional outbox pattern and why it beats dual writes.

Practice

Build a tiny outbox relay: poll outbox rows not yet published, publish them, mark them published, and make a Redis consumer idempotent by event ID.

Trade-offs

↔
Outbox/CDC adds infrastructure and latency (usually sub-second) in exchange for correctness and replayability.

Done when you can

I can place data in PostgreSQL, Kafka and Redis by role.
I avoid dual writes with the outbox or CDC and make Redis consumers idempotent.

Command Palette