Command Palette

Search for a command to run...

Unit 0.3 · Foundations of State

Backups, Restores, and Disaster Recovery

RPO and RTO, snapshots vs logical vs continuous backups, point-in-time recovery, the 3-2-1 rule, Velero for Kubernetes, and why an untested backup doesn't count.

Intermediate 40 min 3 lab steps 2 failure drills

Start here

The mental model

Replication protects you from a machine dying. It does NOT protect you from DELETE FROM orders without a WHERE: the replicas faithfully delete it too, within milliseconds. Backups protect you from mistakes, bugs, ransomware, and bad migrations, because they're copies from the past.

Two numbers define every backup plan. RPO (Recovery Point Objective): how much data can you lose, measured in time? RTO (Recovery Time Objective): how long can you be down while restoring? 'Nightly backups' means an RPO of up to 24 hours. If the business says 5 minutes, you need continuous backups.

Go deeper

How it works inside

01Three kinds of backup

LOGICAL backups export data as SQL or JSON (pg_dump, mongodump, Elasticsearch reindex). They're portable across versions and good for single tables, but slow to restore for large databases (every index is rebuilt). PHYSICAL backups copy the data files or volume blocks (EBS snapshots, pg_basebackup), which is fast to restore but tied to the engine version. CONTINUOUS backups ship the transaction log (Postgres WAL archiving with pgBackRest or WAL-G, RDS automated backups, Kafka MirrorMaker to another cluster) and allow POINT-IN-TIME RECOVERY (PITR): restore the last base backup, replay the log up to 14:02:59, one second before the bad DELETE.

Three kinds of backupdiagram
Rendering diagram…

02The 3-2-1 rule and immutability

Keep at least 3 copies, on 2 different media or services, with 1 off-site (another region or account). Modern addition: 1 IMMUTABLE copy that nobody (including an attacker with admin credentials) can delete, such as S3 Object Lock in compliance mode or AWS Backup Vault Lock in a separate account. Ransomware attacks deliberately delete backups first (AWS course, multi-account security baseline).

03Kubernetes: Velero

VELERO backs up Kubernetes objects (as JSON to S3) and persistent volumes (as cloud snapshots or file-level copies with Kopia) on a schedule, and restores them into the same or another cluster. With GitOps you can recreate the objects from Git, so Velero's main value is the VOLUMES and anything not in Git. For databases, prefer the database's own consistent backup (pgBackRest, operator-managed backups) over volume snapshots of a running database, or at least use pre-backup hooks to freeze writes.

04Restore testing

A backup is only proven by a restore. Automate it: a weekly job restores last night's backup into a scratch instance, runs sanity queries (row counts, latest order timestamp), records how long it took (your real RTO), and alerts if anything fails. Teams that skip this discover during an incident that backups were empty, encrypted with a lost key, or take 19 hours to restore (SRE course, game days).

Do it

Hands-on lab

  1. 1

    Logical backup and restore with pg_dump

    Start Postgres in Docker, create data, dump it in custom format (compressed, parallel-restorable), drop the table, and restore.

    terminal
    $ docker run -d --name pg -e POSTGRES_PASSWORD=pw -p 5432:5432 postgres:17
    export PGHOST=localhost PGUSER=postgres PGPASSWORD=pw
    psql -c "create table orders(id serial primary key, total numeric, created_at timestamptz default now())"
    psql -c "insert into orders(total) select random()*100 from generate_series(1,100000)"
    pg_dump -Fc -f shop.dump postgres && ls -lh shop.dump
    psql -c 'drop table orders'
    pg_restore -d postgres shop.dump && psql -c 'select count(*) from orders'
    ── expected output ──
    -rw-r--r-- 1 you you 2.1M shop.dump
    count
    --------
    100000
  2. 2

    Point-in-time recovery on RDS

    RDS keeps automated backups plus transaction logs for the retention period (1–35 days), so you can restore to any second. A restore always creates a NEW instance; you then repoint the app or copy the missing rows back. Terraform course Stage 4 created this database with backup_retention_period set.

    terminal
    $ aws rds describe-db-instances --db-instance-identifier shoplite-prod --query 'DBInstances[0].LatestRestorableTime'
    aws rds restore-db-instance-to-point-in-time \
    --source-db-instance-identifier shoplite-prod \
    --target-db-instance-identifier shoplite-prod-pitr \
    --restore-time 2026-09-27T08:32:59Z
    ── expected output ──
    "2026-09-27T08:41:10+00:00"
    { "DBInstance": { "DBInstanceIdentifier": "shoplite-prod-pitr", "DBInstanceStatus": "creating", ... } }
  3. 3

    Velero: back up a namespace, restore it elsewhere

    After installing Velero with an S3 bucket and snapshot provider, a scheduled backup runs daily. Restoring into a new namespace is a safe way to test.

    terminal
    $ velero schedule create shop-daily --schedule '0 2 * * *' --include-namespaces shoplite-prod --ttl 720h
    velero backup create shop-now --from-schedule shop-daily --wait
    velero restore create --from-backup shop-now --namespace-mappings shoplite-prod:shoplite-restore-test --wait
    kubectl -n shoplite-restore-test get pods,pvc
    ── expected output ──
    Backup completed with status: Completed.
    Restore completed with status: Completed.
    NAME READY STATUS
    pod/web-7c9d8e6f5-abcde 1/1 Running
    ...
    persistentvolumeclaim/data-redis-0 Bound

Operate it

Knobs that matter

SettingDefaultWhat it doesWhen to change it
backup_retention_period (RDS)7 days (console) / 1 (API)How far back PITR can go.Set explicitly (7–35) and match the business's recovery needs; 0 disables automated backups.
archive_timeout (Postgres)0 (off)Forces a WAL segment switch after N seconds so it gets archived even when traffic is low.Set to 60s to bound RPO on quiet databases when using WAL archiving.
Velero --ttl720hHow long a backup is kept before garbage collection.Align with retention policy; keep monthly backups longer via a separate schedule.
S3 Object LockoffMakes objects undeletable for a period.Enable (compliance mode) on the backup bucket in a separate account for ransomware resilience.

3am practice

Failure drills

Each drill is a real failure mode. Read the scenario and the output, decide what's wrong, then reveal the diagnosis.

Drill #1

The backup that never ran

A developer drops a table in prod. You go to restore last night's dump from S3.

terminal
$ aws s3 ls s3://shoplite-backups/pg/ --recursive | tail -3
── what you'll see ──
2026-06-02 02:00:14 2147318 pg/shop-2026-06-02.dump
2026-06-03 02:00:11 2147902 pg/shop-2026-06-03.dump
2026-06-04 02:00:09 0 pg/shop-2026-06-04.dump

Drill #2

Restoring takes longer than the outage budget

Your 800 GB database needs restoring from a logical pg_dump. The SLO allows 1 hour of downtime.

terminal
$ pg_restore -j 8 -d shop shop.dump # started 10:05
── what you'll see ──
... (11 hours later) ...
pg_restore: creating INDEX "public.orders_created_at_idx"

The bigger picture

Connects to

Prove it

Interview questions

01

What are RPO and RTO? Give an example of meeting a 5-minute RPO.

02

Why isn't replication a backup?

03

How do you know your backups work?

0/3 · 0%