Topic 10.3
Malware Hashes and Global Blocklists
In one line
For 1B known-malicious file hashes, a local filter says "definitely not known-bad" for almost every upload in microseconds; "maybe" is confirmed against the authoritative store. A 5B-entry global blocklist with sub-millisecond checks uses local static filters per node, updated via versioned snapshots and deltas, with a fail-closed or quarantine policy.
Think of it like this
Airport security with a watch list. Officers have a compact card that rules out almost every traveller instantly; anyone flagged "maybe" is checked against the full database before any action.
Key ideas
- 01
Hash normalisation: compare the same digest type (SHA-256 of the raw file), canonical encoding (lowercase hex or raw 32 bytes). Different encodings of the same hash would be false negatives.
- 02
False positives mean an extra authoritative lookup (not a block): acceptable. False negatives mean malware passes: unacceptable, so updates must be complete and fast.
- 03
Static filters fit blocklists: sets change by batch, so xor or binary fuse filters (~20–30% smaller than Bloom) rebuilt periodically, plus a small dynamic Bloom filter for entries added since the last build.
- 04
Distribution: central builder → versioned snapshot → regional object storage → nodes; deltas every minute via a pub/sub topic; nodes report version; alert on stale nodes.
- 05
Failure policy: if a node's filter is missing or too old, fail to the authoritative service, or quarantine uploads for later scanning; never silently fail open for security.
Code & diagrams
Interview problem
The problem
Malware hash lookup for 1B hashes, and a 5B global blocklist
(1) Every uploaded file must be checked against 1 billion known malicious hashes; design the filter plus authoritative store, covering memory, false positives, hash normalisation, security and updates. (2) A 5B-entry blocklist must be checked in under 1 ms locally in four regions; design local filters, periodic updates, versioning, security and failover.
Explain it without notes
Why are static filters (xor, binary fuse) attractive for blocklists?
Practice
Compute authoritative lookups per second for 20K uploads/sec at FPR 0.1% and 0.01%.
Trade-offs
- ↔
Local filters give sub-millisecond checks but need a robust distribution pipeline and a clear failure policy.
Done when you can
I can design blocklist filters with complete updates, versioning and security-appropriate failure behaviour.