Replay, Reprocessing, and Backfills

In one line

Replaying events fixes bugs and builds new views: reset a consumer group's offsets (or start a new group) to a time or offset and reprocess. The danger is side effects: replays must not resend emails, recharge cards or double-count, so separate pure projections from side-effecting consumers and rely on idempotency.

0/5 · 0%

Think of it like this

Re-marking exam papers after finding a mistake in the answer key. You re-read every paper and fix the grades, but you don't re-send every student their certificate or refund their exam fees.

Key ideas

01
Mechanisms: kafka-consumer-groups --reset-offsets --to-datetime 2026-09-18T00:00:00.000 --execute (group must be stopped), --to-earliest, --shift-by, or a brand-new group ID starting from a chosen point; Kafka Streams apps use the application reset tool.
02
New group vs reset: a new group (for example billing-projector-v2) builds output side by side, letting you compare before switching; resetting the existing group overwrites in place.
03
Side effects: split consumers into pure projections (safe to replay) and side-effect handlers (emails, payments, webhooks). Replays should target only the former, or run with side effects disabled (a flag), and idempotency protects the rest.
04
Load: replays read old data from disk (or tiered storage), competing with live traffic. Throttle with consumer quotas and run during quiet hours.
05
Limits: you can only replay what's retained. For long-lived reprocessing needs, keep compacted state topics, long retention with tiered storage, or archives in a data lake.

Code & diagrams

replay.shbash

# 1. Stop the consumer deployment (group must be inactive for a reset)
kubectl scale deploy billing-projector --replicas=0

# 2. Preview, then reset to the first bad day
kafka-consumer-groups.sh --bootstrap-server $B --group billing-projector \
  --topic commerce.orders --reset-offsets --to-datetime 2026-09-18T00:00:00.000 --dry-run
kafka-consumer-groups.sh --bootstrap-server $B --group billing-projector \
  --topic commerce.orders --reset-offsets --to-datetime 2026-09-18T00:00:00.000 --execute

# 3. Start the fixed version with side effects disabled and a fetch quota
kubectl set env deploy/billing-projector SIDE_EFFECTS_ENABLED=false
kubectl scale deploy billing-projector --replicas=6

Interview problem

The problem

A consumer processed 10 days of events incorrectly

A bug made a consumer compute wrong billing totals for 10 days. The bug is fixed. How do you rebuild correct state? Discuss new consumer groups, offset reset, replay, side effects and idempotency.

Explain it without notes

Why build a replay into a new consumer group and output instead of resetting in place?

Practice

Reset a lab consumer group to a timestamp with --dry-run, then execute, and verify reprocessing starts from the right offsets.

Trade-offs

↔
Replays are Kafka's superpower for correcting mistakes, but only with retained data and side-effect discipline.

Done when you can

I can plan a safe replay or backfill without repeating side effects.

Command Palette