The Cache Stampede and How to Never Have It Again

The scene: an item goes viral. 10,000 requests for the same key arrive in the same second. The cache misses. Ten thousand threads all call the database. The cache was supposed to protect the DB — and it became the attack.

The mechanics

// NAIVE cache-aside — the stampede in one function
public Object get(String key) {
  Object v = redis.get(key);
  if (v != null) return v;              // hit
  v = db.load(key);                     // 10,000 threads land here together
  redis.set(key, v, TTL);
  return v;
}

// FIX — one flight per key: losers wait on the winner's Future
public CompletableFuture<Object> get(String key) {
  return flights.computeIfAbsent(key, k ->
    CompletableFuture.supplyAsync(() -> {
      Object v = redis.get(key);
      if (v != null) return v;
      Object loaded = db.load(key);     // exactly ONE loader
      redis.set(key, loaded, TTL);
      return loaded != null ? loaded : NULL_PLACEHOLDER; // don't cache the Future
    }));
}

The three-part vaccine

• —Single-flight: one in-flight load per key; everyone else waits on the same Future (the code above).
• —TTL jitter: a fixed TTL makes all copies of a hot key expire together — add a random offset so expiry is spread out.
• —Cache the null: cache misses for 60 seconds so a torrent of 'no such id' requests doesn't hit the DB either.

The distributed version (Redis, honestly)

Single-flight held in JVM memory works per instance but not across instances — five app nodes each start their own load. For a cross-node flight, use Redis SETNX as a distributed lock around the load, or rely on the DB-level protection (the query is the same whether 1 or 100 run it). Realistically, jittered TTL + a hot-key cache at the instance level gets you 95% of the way.

Caches don't fail gradually. They fail at the exact moment 10,000 requests agree to miss at once.