Why State Is Hard: Durability, Replication, Consistency

01Durability: when is a write really saved?

When a program writes to a file, the data first lands in the operating system's page cache in RAM, not on disk. If the machine loses power, it's gone. fsync() forces it to the physical device and waits (Linux course, disk and storage). Databases make a write durable by appending it to a WRITE-AHEAD LOG (WAL; Kafka calls it the log, Redis calls it the AOF) and fsyncing that log before acknowledging the client. The real data files are updated later, and after a crash the log is replayed.

fsync is slow (milliseconds on network disks like EBS), so systems let you trade safety for speed: Postgres synchronous_commit, Redis appendfsync everysec, Kafka relying on replication instead of per-message fsync. Knowing where each system sits on that dial is half of operating it.

02Replication: copies on other machines

One machine can always die, so data is copied to others. LEADER–FOLLOWER (primary–replica) replication is the most common: all writes go to one leader, which streams its log to followers (Postgres, MySQL, Redis, Kafka per partition). MULTI-LEADER and LEADERLESS designs (Cassandra, DynamoDB) accept writes anywhere and reconcile later.

SYNCHRONOUS replication waits for a follower to confirm before acknowledging the client: no data loss if the leader dies, but every write pays a network round-trip and a slow follower slows everyone. ASYNCHRONOUS replication acknowledges immediately: fast, but the last few writes can be lost on failover, and followers LAG (reads from them can be stale). Most production setups are 'semi-synchronous': at least one follower must confirm.

03Consistency, CAP, and PACELC

CAP says that during a network PARTITION (nodes can't talk to each other), a system must choose between CONSISTENCY (every read sees the latest write, so it refuses requests it can't guarantee) and AVAILABILITY (it answers anyway, possibly with stale data). You can't opt out of partitions, so the real choice is CP or AP during failures.

PACELC adds the everyday case: Else (no partition), you still trade LATENCY against CONSISTENCY. Waiting for replicas to agree takes time. Postgres with synchronous replicas is PC/EC; DynamoDB by default is PA/EL (eventually consistent reads) with an option for strongly consistent reads.

04Quorums

With N copies, if a write must reach W of them and a read must ask R of them, then W + R > N guarantees the read overlaps at least one copy with the latest write. With N=3, W=2, R=2 you tolerate one node failure for both reads and writes. Kafka's min.insync.replicas=2 with acks=all and a replication factor of 3 is exactly this idea. Leader election uses a MAJORITY quorum too (Raft in etcd and Kafka KRaft), which is why clusters have odd numbers of nodes: 3 nodes tolerate 1 failure, and 4 still tolerate only 1.