Unit 0.2 · Foundations of State

Running Stateful Workloads: Disks, Kubernetes, Operators, Managed Services

Where stateful systems actually run: disk types and IOPS, StatefulSets and persistent volumes, operators that automate day-2 work, and when to just pay for a managed service.

Intermediate 45 min 3 lab steps 2 failure drills

Start here

The mental model

A stateless pod is cattle: kill it and a new one appears anywhere. A database pod is a pet with a name and its own bed: postgres-0 must come back as postgres-0, on the same data, and never two of them at once. Kubernetes can do that (StatefulSets and persistent volumes), but the harder parts, like failover, backups, upgrades, and resizing, need either a human, an OPERATOR (software that encodes the human's runbook), or a MANAGED SERVICE (someone else's operators and on-call team).

Go deeper

How it works inside

01Disks: IOPS, throughput, latency

Databases care about three disk numbers. IOPS is operations per second (small random reads and writes: index lookups, WAL fsyncs). THROUGHPUT is MB/s (big sequential scans, Kafka log segments, backups). LATENCY is time per operation, and p99 fsync latency directly limits commit rate.

On AWS (AWS course, EBS/EFS topic): gp3 gives 3,000 IOPS and 125 MB/s baseline, adjustable independently, and is the right default. io2 gives provisioned high IOPS with low latency for busy databases. Local NVMe instance store is fastest but EPHEMERAL: data is lost when the instance stops, so only use it for systems that replicate themselves (Kafka, OpenSearch, Cassandra) and can rebuild a node from peers. Never run databases on EFS/NFS; network file locking and fsync semantics make it slow and risky.

02StatefulSets and volumes

A StatefulSet gives each replica a stable name (kafka-0, kafka-1), a stable DNS entry through a headless Service (kafka-0.kafka.shop.svc), ordered startup and shutdown, and its own PersistentVolumeClaim from volumeClaimTemplates (Kubernetes course, StatefulSets and PV/PVC topics). If kafka-1 is rescheduled, it gets the same claim back.

Gotchas: EBS volumes live in one availability zone, so a pod whose volume is in ap-south-1a can only run on nodes in that zone (use WaitForFirstConsumer StorageClasses and spread replicas across zones). Deleting a StatefulSet does NOT delete its PVCs, which is a safety feature, and also means orphaned disks you keep paying for. allowVolumeExpansion: true lets you grow a PVC online.

StatefulSets and volumesdiagram

Rendering diagram…

03Operators: runbooks as code

An OPERATOR is a controller plus CRDs that manages a specific system (Kubernetes course, CRDs and operators). You declare kind: Kafka with 3 brokers, or kind: Cluster with 2 Postgres replicas and nightly backups to S3, and the operator creates the StatefulSets, handles failover, rolling upgrades, certificate rotation, and backups. Mature ones: CloudNativePG and Zalando/Crunchy for Postgres, Strimzi for Kafka, the OpenSearch Operator, and the Redis operators (quality varies).

04Managed vs self-hosted

RDS/Aurora, Amazon MSK, ElastiCache/MemoryDB, and Amazon OpenSearch Service run the same engines for you: patching, backups, failover, and monitoring included. You pay more per GB/hour and give up some configuration and version control. Self-hosting (on VMs or Kubernetes) is cheaper at scale and more flexible, but you own every upgrade and every 3am failover.

A useful default for most teams: MANAGED for the primary database and anything you can't afford to lose; self-host on Kubernetes when you have a platform team, a good operator, and a real reason (cost at scale, features, multi-cloud). Stateless things belong on Kubernetes; state goes there deliberately, not by default.

Do it

Hands-on lab

A StatefulSet with per-pod storage

On any cluster with a default StorageClass (kind has one). Each pod writes its own hostname into its own volume.

sts-demo.yamlwhole fileyaml

apiVersion: v1
kind: Service
metadata: { name: demo }
spec:
  clusterIP: None          # headless: DNS per pod
  selector: { app: demo }
  ports: [{ port: 80 }]
---
apiVersion: apps/v1
kind: StatefulSet
metadata: { name: demo }
spec:
  serviceName: demo
  replicas: 3
  selector: { matchLabels: { app: demo } }
  template:
    metadata: { labels: { app: demo } }
    spec:
      containers:
        - name: app
          image: busybox:1.37
          command: ["sh", "-c", "hostname >> /data/boots; sleep infinity"]
          volumeMounts: [{ name: data, mountPath: /data }]
  volumeClaimTemplates:
    - metadata: { name: data }
      spec:
        accessModes: [ReadWriteOnce]
        resources: { requests: { storage: 100Mi } }

terminal

$ kubectl apply -f sts-demo.yaml && kubectl rollout status sts/demo

kubectl get pvc

── expected output ──

partitioned roll out complete: 3 new pods have been updated...

NAME STATUS VOLUME CAPACITY

data-demo-0 Bound pvc-3b1f... 100Mi

data-demo-1 Bound pvc-8c2e... 100Mi

data-demo-2 Bound pvc-d41a... 100Mi

2
Kill a pod: same name, same disk
The replacement is demo-1 again, it mounts data-demo-1 again, and its file now has two boot lines.
terminal
$ kubectl delete pod demo-1 && kubectl wait --for=condition=Ready pod/demo-1
kubectl exec demo-1 -- cat /data/boots
── expected output ──
demo-1
demo-1
3
Delete the StatefulSet: the data stays
The PVCs survive. Recreate the StatefulSet and each pod reattaches its old data. Clean up the PVCs explicitly when you really mean it.
terminal
$ kubectl delete sts demo && kubectl get pvc --no-headers | wc -l
kubectl delete pvc data-demo-0 data-demo-1 data-demo-2
── expected output ──
3

Operate it

Knobs that matter

Setting	Default	What it does	When to change it
volumeBindingMode	Immediate (some classes)	When the volume is created: at claim time or when a pod is scheduled.	Use `WaitForFirstConsumer` so the disk is created in the same zone as the pod.
reclaimPolicy	Delete	What happens to the underlying disk when the PV is released.	Use `Retain` for production databases so a deleted PVC doesn't destroy the disk.
podManagementPolicy	OrderedReady	Start and stop pods one at a time in order.	`Parallel` for systems that don't need ordering (Kafka with KRaft, OpenSearch data nodes) to speed up restarts.
PodDisruptionBudget	none	Limits voluntary evictions (node drains, upgrades).	Always set `maxUnavailable: 1` for quorum systems so a drain never takes down two members at once.
gp3 IOPS / throughput	3000 / 125 MB/s	Provisioned performance of an EBS gp3 volume.	Raise independently of size when `iostat` shows the device saturated (`%util` near 100, rising `await`).

3am practice

Failure drills

Each drill is a real failure mode. Read the scenario and the output, decide what's wrong, then reveal the diagnosis.

Drill #1

Pod stuck Pending after a node replacement

During a cluster upgrade, postgres-1 is rescheduled and never starts.

terminal

$ kubectl describe pod postgres-1 | tail -3

── what you'll see ──

Warning FailedScheduling 0/6 nodes are available: 3 node(s) had volume node affinity conflict, 3 node(s) didn't match Pod's node affinity/selector.

Drill #2

Two brokers down during a routine drain

An engineer drains two nodes at once for patching. Kafka producers with acks=all start failing.

terminal

$ kafka-topics.sh --bootstrap-server kafka:9092 --describe --under-min-isr-partitions

── what you'll see ──

Topic: orders Partition: 4 Leader: 0 Replicas: 0,1,2 Isr: 0

Topic: orders Partition: 7 Leader: 0 Replicas: 2,0,1 Isr: 0

Decide

Where should this stateful system run?

Option	You manage	Good for	Watch out for
Managed service (RDS, MSK, ElastiCache, OpenSearch Service)	Schema, sizing, parameters	Primary databases, small teams, compliance	Cost at scale, version lag, limited tuning
Kubernetes + mature operator	Operator config, storage, upgrades of the operator	Platform teams, many similar clusters, portability	Operator bugs, AZ-bound volumes, needs strong K8s skills
VMs + config management (Ansible)	Everything	Legacy estates, very specific tuning	Manual failover and patching unless automated

The bigger picture

Connects to

System Design · Stateless Services

The principle that makes horizontal scaling, rolling deploys, and instance-juggling boring.

Kubernetes · Volumes, PV & PVC

The claim/volume model this unit builds on.

Kubernetes · StorageClasses

Dynamic provisioning, binding modes, and reclaim policies.

Kubernetes · CRDs & Operators

How operators like CloudNativePG and Strimzi work under the hood.

AWS · EBS & EFS

Volume types, IOPS, snapshots, and why EFS isn't for databases.

Prove it

Interview questions

Would you run a production database on Kubernetes?

What does a StatefulSet give you that a Deployment doesn't?

0/3 · 0%

Command Palette

The mental model

How it works inside

01Disks: IOPS, throughput, latency

02StatefulSets and volumes

03Operators: runbooks as code

04Managed vs self-hosted

Hands-on lab

A StatefulSet with per-pod storage

Kill a pod: same name, same disk

Delete the StatefulSet: the data stays

Knobs that matter

Failure drills

Pod stuck Pending after a node replacement

Two brokers down during a routine drain

Where should this stateful system run?

Connects to

Interview questions