Unit 0.3 · Foundations of State
Backups, Restores, and Disaster Recovery
RPO and RTO, snapshots vs logical vs continuous backups, point-in-time recovery, the 3-2-1 rule, Velero for Kubernetes, and why an untested backup doesn't count.
Start here
The mental model
Replication protects you from a machine dying. It does NOT protect you from DELETE FROM orders without a WHERE: the replicas faithfully delete it too, within milliseconds. Backups protect you from mistakes, bugs, ransomware, and bad migrations, because they're copies from the past.
Two numbers define every backup plan. RPO (Recovery Point Objective): how much data can you lose, measured in time? RTO (Recovery Time Objective): how long can you be down while restoring? 'Nightly backups' means an RPO of up to 24 hours. If the business says 5 minutes, you need continuous backups.
Go deeper
How it works inside
01Three kinds of backup
LOGICAL backups export data as SQL or JSON (pg_dump, mongodump, Elasticsearch reindex). They're portable across versions and good for single tables, but slow to restore for large databases (every index is rebuilt). PHYSICAL backups copy the data files or volume blocks (EBS snapshots, pg_basebackup), which is fast to restore but tied to the engine version. CONTINUOUS backups ship the transaction log (Postgres WAL archiving with pgBackRest or WAL-G, RDS automated backups, Kafka MirrorMaker to another cluster) and allow POINT-IN-TIME RECOVERY (PITR): restore the last base backup, replay the log up to 14:02:59, one second before the bad DELETE.
02The 3-2-1 rule and immutability
Keep at least 3 copies, on 2 different media or services, with 1 off-site (another region or account). Modern addition: 1 IMMUTABLE copy that nobody (including an attacker with admin credentials) can delete, such as S3 Object Lock in compliance mode or AWS Backup Vault Lock in a separate account. Ransomware attacks deliberately delete backups first (AWS course, multi-account security baseline).
03Kubernetes: Velero
VELERO backs up Kubernetes objects (as JSON to S3) and persistent volumes (as cloud snapshots or file-level copies with Kopia) on a schedule, and restores them into the same or another cluster. With GitOps you can recreate the objects from Git, so Velero's main value is the VOLUMES and anything not in Git. For databases, prefer the database's own consistent backup (pgBackRest, operator-managed backups) over volume snapshots of a running database, or at least use pre-backup hooks to freeze writes.
04Restore testing
A backup is only proven by a restore. Automate it: a weekly job restores last night's backup into a scratch instance, runs sanity queries (row counts, latest order timestamp), records how long it took (your real RTO), and alerts if anything fails. Teams that skip this discover during an incident that backups were empty, encrypted with a lost key, or take 19 hours to restore (SRE course, game days).
Do it
Hands-on lab
- 1
Logical backup and restore with pg_dump
Start Postgres in Docker, create data, dump it in custom format (compressed, parallel-restorable), drop the table, and restore.
terminal$ docker run -d --name pg -e POSTGRES_PASSWORD=pw -p 5432:5432 postgres:17export PGHOST=localhost PGUSER=postgres PGPASSWORD=pwpsql -c "create table orders(id serial primary key, total numeric, created_at timestamptz default now())"psql -c "insert into orders(total) select random()*100 from generate_series(1,100000)"pg_dump -Fc -f shop.dump postgres && ls -lh shop.dumppsql -c 'drop table orders'pg_restore -d postgres shop.dump && psql -c 'select count(*) from orders'── expected output ──-rw-r--r-- 1 you you 2.1M shop.dumpcount--------100000 - 2
Point-in-time recovery on RDS
RDS keeps automated backups plus transaction logs for the retention period (1–35 days), so you can restore to any second. A restore always creates a NEW instance; you then repoint the app or copy the missing rows back. Terraform course Stage 4 created this database with
backup_retention_periodset.terminal$ aws rds describe-db-instances --db-instance-identifier shoplite-prod --query 'DBInstances[0].LatestRestorableTime'aws rds restore-db-instance-to-point-in-time \--source-db-instance-identifier shoplite-prod \--target-db-instance-identifier shoplite-prod-pitr \--restore-time 2026-09-27T08:32:59Z── expected output ──"2026-09-27T08:41:10+00:00"{ "DBInstance": { "DBInstanceIdentifier": "shoplite-prod-pitr", "DBInstanceStatus": "creating", ... } } - 3
Velero: back up a namespace, restore it elsewhere
After installing Velero with an S3 bucket and snapshot provider, a scheduled backup runs daily. Restoring into a new namespace is a safe way to test.
terminal$ velero schedule create shop-daily --schedule '0 2 * * *' --include-namespaces shoplite-prod --ttl 720hvelero backup create shop-now --from-schedule shop-daily --waitvelero restore create --from-backup shop-now --namespace-mappings shoplite-prod:shoplite-restore-test --waitkubectl -n shoplite-restore-test get pods,pvc── expected output ──Backup completed with status: Completed.Restore completed with status: Completed.NAME READY STATUSpod/web-7c9d8e6f5-abcde 1/1 Running...persistentvolumeclaim/data-redis-0 Bound
Operate it
Knobs that matter
| Setting | Default | What it does | When to change it |
|---|---|---|---|
| backup_retention_period (RDS) | 7 days (console) / 1 (API) | How far back PITR can go. | Set explicitly (7–35) and match the business's recovery needs; 0 disables automated backups. |
| archive_timeout (Postgres) | 0 (off) | Forces a WAL segment switch after N seconds so it gets archived even when traffic is low. | Set to 60s to bound RPO on quiet databases when using WAL archiving. |
| Velero --ttl | 720h | How long a backup is kept before garbage collection. | Align with retention policy; keep monthly backups longer via a separate schedule. |
| S3 Object Lock | off | Makes objects undeletable for a period. | Enable (compliance mode) on the backup bucket in a separate account for ransomware resilience. |
3am practice
Failure drills
Each drill is a real failure mode. Read the scenario and the output, decide what's wrong, then reveal the diagnosis.
Drill #1
The backup that never ran
A developer drops a table in prod. You go to restore last night's dump from S3.
Drill #2
Restoring takes longer than the outage budget
Your 800 GB database needs restoring from a logical pg_dump. The SLO allows 1 hour of downtime.
The bigger picture
Connects to
System Design · Disaster Recovery
RPO, RTO, backup and restore at the REGION level — the plan for when the whole datacenter dies.
AWS · RDS & Aurora
Automated backups, snapshots, Multi-AZ, and read replicas.
AWS · S3
Where backups live: versioning, lifecycle rules, Object Lock.
SRE · Game day
Practise a restore under realistic pressure.
GitOps · Running GitOps in production
Rebuilding clusters from Git, and what Git doesn't contain (data).
Prove it
Interview questions
What are RPO and RTO? Give an example of meeting a 5-minute RPO.
Why isn't replication a backup?
How do you know your backups work?