Stale Filters and Insert Consistency

In one line

Staleness in one direction is harmless (the filter has extra items → false positives); in the other it breaks correctness (the database has an item the filter lacks → false negatives, valid requests rejected). Order updates so an item is in the filter before it's exposed, make updates reliable (outbox/CDC, retries), and give recent items a fallback path.

0/4 · 0%

Think of it like this

A building's visitor list at reception. If it still lists someone who left, reception sends a visitor up needlessly (annoying, harmless). If it's missing a new employee, reception turns away someone who should get in (harmful). Only the second mistake needs urgent prevention.

Key ideas

01
Harmless staleness: items deleted from the database but still in the filter; the database answers "not found", costing one lookup.
02
Dangerous staleness: items in the database but not in the filter; the filter says "definitely absent" and the request is rejected without asking the database. This is a false negative the application created.
03
How it happens: database insert succeeds and filter update fails (network error, Redis down, crash between the two); asynchronous update lag; a rebuild that missed a window of inserts; a filter restored from an old snapshot without replaying changes.
04
Ordering: add to the filter before the item becomes visible, or at least before it can be requested (for example add to filter, then commit and publish), accepting that a failed database insert leaves an extra filter bit (harmless).
05
Reliability: outbox or CDC-driven updates with retries (at-least-once; adding an item twice is idempotent), and a periodic full rebuild as a safety net.
06
Fallback for recency: if IDs encode creation time (or you can cheaply know it), treat IDs newer than the filter's snapshot position as "maybe" and verify with the database.

Code & diagrams

directions.mermaiddiagram

Rendering diagram…

safe-insert.javajava

@Transactional
public Product create(NewProduct cmd) {
    Product p = repo.save(Product.from(cmd));
    outbox.save(OutboxEvent.of("ProductCreated", p.id()));   // same DB transaction
    return p;
}
// Relay/CDC publishes ProductCreated -> filter updater does bf.add(id) (idempotent, retried).
// Reads: if id > filterManifest.maxIdCovered -> skip the filter, ask the DB (recent-item fallback).

Interview problem

The problem

DB insert succeeds, filter update fails, user queries the item

T1: an item is inserted into the database. T2: the Bloom filter update fails. T3: a user queries the item and the filter says "definitely absent". Design a recovery and prevention strategy covering the source of truth, rebuild, repair, update ordering and failure recovery.

When it breaks

Filter updates done with fire-and-forget calls after the DB commit

What you see

Occasional network errors silently drop updates; a small, growing fraction of valid items are rejected as nonexistent.

Fix & prevent

Outbox/CDC with retries, recent-item fallback, periodic rebuilds, and sampled negative verification.

Explain it without notes

Which direction of staleness is dangerous and why?

Practice

Design a sampled check that detects dangerous staleness in production.

Trade-offs

↔
Reliable pipelines cost infrastructure; recent-item fallbacks cost some database lookups; both are cheaper than rejecting valid requests.

Done when you can

I can keep a filter consistent with its source of truth and detect dangerous staleness.

Command Palette