Command Palette

Search for a command to run...

Hectal
Phase 8Advanced9 of 12 in Bloom Filters

Distributed Bloom Filters

Event-driven filter updates with an outbox, CDC and Kafka; per-shard filters that cut query fan-out; filters under consistent-hashing rebalances; and multi-region filters with failover.

At scale, filters are fed by change streams, spread across many nodes, split by shard and replicated across regions. The core rule stays the same: the filter must never miss an item the source of truth has, even while it lags, moves or fails over.

0/3 · 0%
3 topics ~25 min 4 code blocks & diagrams
Start with the first topic
1
8.1

Reliable Updates with Outbox, CDC, and Kafka

Feed filters from the database's change stream: an outbox row or CDC event per insert goes to Kafka; a filter-updater consumer group applies idempotent adds to Redis or to local filters in every instance. You get retries, replay for rebuilds, multiple consumers and monitoring through consumer lag.

9 min 1 diagram 1 code practice

2
8.2

Per-Shard Filters for Routing and Fan-Out Reduction

When a key could live on any of N shards (no deterministic routing), keep a Bloom filter per shard and query only shards whose filter says maybe. With 100 shards at 1% FPR, a lookup touches about 2 shards instead of 100. Shard moves and rebalances require filter updates, double-writes or versioned rebuilds.

8 min 1 diagram practice

3
8.3

Multi-Region Filters and Failover

Each region keeps its own filters for low-latency checks, fed by replicated change streams (Kafka mirroring or CDC per region) and versioned snapshots. Regional filters are eventually consistent; recent-item fallback, residency rules and a failover plan (rebuild or load a peer region's snapshot) keep them correct.

8 min 1 diagram practice