Unit 3.3 · Redis and Caching

Replication, Sentinel, Cluster, Pub/Sub, and Distributed Locks

Async replication and its data-loss window, Sentinel failover, Redis Cluster hash slots and MOVED redirects, hash tags, pub/sub vs streams, and how (not) to build a distributed lock.

Advanced 55 min 4 lab steps 2 failure drills

Start here

The mental model

Three ways to grow Redis. REPLICATION: copies for read scaling and fast recovery. SENTINEL: watchdogs that notice when the primary dies and promote a replica (high availability for one dataset). CLUSTER: split the data into 16,384 slots across several primaries so the dataset and throughput can exceed one machine (sharding), each primary with its own replicas.

A distributed lock in Redis is a sticky note that says 'mine until 10:00:30'. It works well for efficiency (avoid doing the same job twice) and dangerously for correctness (guarantee two workers NEVER overlap), because clocks, pauses, and failovers can let two workers both believe they hold it.

Go deeper

How it works inside

01Replication

Replicas connect with REPLICAOF host port, receive a full RDB snapshot, then a stream of write commands. Replication is ASYNCHRONOUS: the primary acknowledges clients before replicas have the write, so a failover can lose the last writes. WAIT numreplicas timeout lets a client block until N replicas acknowledged, which reduces but doesn't eliminate that window. min-replicas-to-write makes an isolated primary stop accepting writes, limiting split-brain loss.

02Sentinel

Run 3+ Sentinel processes (on different machines or zones). They monitor the primary; when a QUORUM agrees it's down, one Sentinel is elected to promote the best replica and reconfigure the others. Clients ask Sentinel 'who is the primary for shop-cache?' rather than hard-coding an address, so your client library must be Sentinel-aware. ElastiCache and MemoryDB (non-cluster mode with replicas and Multi-AZ) provide the same behaviour, managed.

03Redis Cluster

Keys map to one of 16,384 HASH SLOTS (CRC16(key) mod 16384), and slots are split across primaries. Clients cache the slot map; ask the wrong node and it replies MOVED 3999 10.0.1.12:6379, and a smart client updates its map. During resharding, ASK redirects handle keys mid-migration.

Multi-key operations (MGET, transactions, Lua scripts touching several keys) only work if all keys are in the SAME slot. HASH TAGS force that: only the part inside {} is hashed, so cart:{user42}:items and cart:{user42}:total land together. Overusing one tag creates a hot slot. Cluster mode has one database (no SELECT 1) and needs cluster-aware clients.

Redis Clusterdiagram

Rendering diagram…

04Pub/Sub vs Streams

PUB/SUB pushes each message to whoever is subscribed right now: no storage, no acknowledgement, no replay. Good for cache-invalidation broadcasts and live notifications where missing one is fine. For anything that must not be lost, use STREAMS with consumer groups (Unit 2.4), or a real broker. In Cluster mode, use SHARDED pub/sub (SSUBSCRIBE) so messages don't broadcast to every node.

05Distributed locks, honestly

The basic lock: SET lock:job42 <random-token> NX PX 30000 (only if absent, auto-expire in 30 s). Release ONLY if you still own it, checking the token atomically in a Lua script, or you might delete someone else's lock after yours expired. Pick the expiry longer than the work, or extend it while working.

The limits: if the worker pauses (a long GC pause, a throttled container, Docker course cgroups) past the expiry, a second worker takes the lock while the first still thinks it holds it. Async replication means a failover can lose a just-acquired lock. Redlock (majority across 5 independent Redis nodes) narrows some gaps but still depends on timing assumptions. For CORRECTNESS, use FENCING TOKENS: the lock service hands out an increasing number, and the protected resource (e.g. the database) rejects writes with an older token. Or use a system built for consensus (etcd, ZooKeeper, or a database row lock with SELECT ... FOR UPDATE).

Do it

Hands-on lab

A primary, a replica, and three Sentinels

A compact Compose file. Sentinels monitor shop with a quorum of 2.

compose.yamlwhole fileyaml

services:
  redis-1: { image: redis:8, command: redis-server --appendonly yes }
  redis-2: { image: redis:8, command: redis-server --appendonly yes --replicaof redis-1 6379 }
  sentinel-1: &s
    image: redis:8
    command: >
      sh -c 'printf "port 26379\nsentinel resolve-hostnames yes\nsentinel monitor shop redis-1 6379 2\nsentinel down-after-milliseconds shop 5000\nsentinel failover-timeout shop 15000\n" > /s.conf && redis-sentinel /s.conf'
  sentinel-2: *s
  sentinel-3: *s

terminal

$ docker compose up -d && sleep 5

docker compose exec sentinel-1 redis-cli -p 26379 SENTINEL get-master-addr-by-name shop

── expected output ──

1) "172.21.0.2"

2) "6379"

2
Kill the primary and watch failover
After down-after-milliseconds the Sentinels agree it's down (+odown), elect a leader, and promote the replica.
terminal
$ docker compose stop redis-1 && sleep 20
docker compose logs sentinel-1 | grep -E 'odown|switch-master'
docker compose exec redis-2 redis-cli ROLE | head -1
── expected output ──
sentinel-1 | +odown master shop 172.21.0.2 6379 #quorum 3/2
sentinel-1 | +switch-master shop 172.21.0.2 6379 172.21.0.3 6379
master
3
A six-node cluster and hash tags
Start six local nodes (for p in 7000 7001 7002 7003 7004 7005; do redis-server --port $p --cluster-enabled yes --cluster-config-file nodes-$p.conf --daemonize yes; done), build the cluster with three primaries and one replica each, then look at slots and cross-slot errors.
terminal
$ redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 --cluster-replicas 1 --cluster-yes
redis-cli -p 7000 CLUSTER KEYSLOT cart:user42:items; redis-cli -p 7000 CLUSTER KEYSLOT 'cart:{user42}:items'
redis-cli -c -p 7000 MSET cart:user42:items 3 cart:user42:total 999
redis-cli -c -p 7000 MSET 'cart:{user42}:items' 3 'cart:{user42}:total' 999
── expected output ──
[OK] All 16384 slots covered.
(integer) 3421
(integer) 9189
(error) CROSSSLOT Keys in request don't hash to the same slot
OK

A lock you can release safely

Acquire with a unique token; release only if the token matches, atomically in Lua.

lock.pywhole filepython

import uuid, redis
r = redis.Redis()
RELEASE = r.register_script("""
if redis.call('GET', KEYS[1]) == ARGV[1] then
  return redis.call('DEL', KEYS[1])
end
return 0
""")

def run_exclusive(name: str, work, ttl_ms: int = 30_000):
    token = str(uuid.uuid4())
    if not r.set(f"lock:{name}", token, nx=True, px=ttl_ms):
        return False                     # someone else is doing it: skip (efficiency lock)
    try:
        work()
    finally:
        RELEASE(keys=[f"lock:{name}"], args=[token])
    return True

Operate it

Knobs that matter

Setting	Default	What it does	When to change it
down-after-milliseconds (Sentinel)	30000	How long a primary must be unreachable before it's considered down.	Lower (5–10 s) for faster failover; too low causes failovers on brief network blips.
min-replicas-to-write / min-replicas-max-lag	0 / 10	Primary refuses writes without enough fresh replicas.	Set to 1 / 10 to limit data written to an isolated primary during split brain.
cluster-node-timeout	15000	When a cluster node is considered failing.	Balance fast failover against false positives, as with Sentinel.
cluster-require-full-coverage	yes	Whole cluster stops serving if any slot is uncovered.	`no` to keep serving healthy slots during a partial outage (a cache usually prefers this).

3am practice

Failure drills

Each drill is a real failure mode. Read the scenario and the output, decide what's wrong, then reveal the diagnosis.

Drill #1

Two workers charged the same card

A payment job uses a Redis lock with a 10 s TTL. During a slow day for the payment provider, a few customers were charged twice.

terminal

$ grep 'order-5521' payments.log

── what you'll see ──

12:00:00.000 worker-1 acquired lock:charge:order-5521

12:00:00.050 worker-1 calling provider...

12:00:10.001 worker-2 acquired lock:charge:order-5521 <- lock expired

12:00:10.040 worker-2 calling provider...

12:00:12.300 worker-1 provider OK, charged

12:00:12.910 worker-2 provider OK, charged

Drill #2

Cache writes lost after failover

After a Sentinel failover, a few seconds of cart updates are missing for some users.

terminal

$ docker compose logs sentinel-1 | grep switch-master

── what you'll see ──

+switch-master shop 172.21.0.2 6379 172.21.0.3 6379

Decide

Scaling options

Setup	Solves	Doesn't solve	Client needs
Single node + persistence	Simple cache/store	Node failure (downtime)	Plain client
Primary + replicas	Read scaling, faster recovery	Automatic failover	Read/write split logic
Sentinel	Automatic failover (HA)	Data bigger than one node	Sentinel-aware client
Cluster	Sharding + HA	Cross-slot multi-key ops (without hash tags)	Cluster-aware client

The bigger picture

Connects to

System Design · Distributed Locking

Mutual exclusion across machines: Redis SETNX, leases, fencing tokens, and the 'lock ≠ atomicity' lesson.

System Design · Consistent Hashing

The ring-based mapping that makes adding/removing nodes move only a tiny slice of keys — the trick behind Redis clusters, Cassandra, and LB affinity.

Unit 0.1 · Split brain & quorums

Sentinel's quorum is the same idea as Raft majorities.

Kubernetes · StatefulSets

How Redis replicas and Sentinels get stable identities in Kubernetes.

SRE · Risky change

Resharding or failing over Redis is a classic risky change to plan.

Prove it

Interview questions

Sentinel vs Cluster?

How would you implement a distributed lock with Redis, and what are its limits?

What are hash tags in Redis Cluster?

0/3 · 0%

Command Palette

The mental model

How it works inside

01Replication

02Sentinel

03Redis Cluster

04Pub/Sub vs Streams

05Distributed locks, honestly

Hands-on lab

A primary, a replica, and three Sentinels

Kill the primary and watch failover

A six-node cluster and hash tags

A lock you can release safely

Knobs that matter

Failure drills

Two workers charged the same card

Cache writes lost after failover

Scaling options

Connects to

Interview questions