Cluster Operations: Upgrades, Drain/Cordon & etcd Backups

In one line

Keeping a cluster healthy over years: cordon and drain nodes for maintenance, upgrade the control plane and nodes one minor version at a time, and back up and restore etcd, the database that holds the entire cluster state.

0/5 · 0%

Think of it like this

Renovating an office floor by floor while the company keeps working. Stop assigning new people to a floor (cordon), move everyone to other floors (drain), renovate (upgrade), reopen it, and keep a copy of the master building plans somewhere safe (etcd backup).

Key ideas

01
CORDON (kubectl cordon node) marks a node unschedulable; DRAIN (kubectl drain node --ignore-daemonsets --delete-emptydir-data) evicts pods, respecting PDBs (Topic 9.2), so they reschedule elsewhere; UNCORDON returns it to service. Managed node groups and Karpenter do this automatically during node replacement.
02
UPGRADES: Kubernetes releases a minor version about every 4 months with roughly 14 months of support. Upgrade one minor version at a time: check deprecated/removed APIs first (kubectl convert, pluto, provider upgrade insights), upgrade add-ons (CNI, CoreDNS, kube-proxy, ingress/gateway controllers) to compatible versions, upgrade the control plane, then nodes (rolling replacement). The kubelet may be at most a few minor versions behind the API server, never ahead.
03
ETCD holds every object in the cluster. On managed services (EKS, GKE, AKS) the provider backs it up; on self-managed clusters (kubeadm) you must: etcdctl snapshot save on a schedule, stored off-cluster, and practise etcdctl snapshot restore. Also keep workloads reproducible from Git (GitOps) and back up persistent volumes separately (Velero: Stateful Systems course, backups).
04
Other routine operations: certificate rotation on kubeadm clusters (kubeadm certs check-expiration), watching control-plane health (API server latency, etcd DB size), and capacity reviews. The DevOps guide on the advanced Kubernetes ecosystem covers blue-green cluster upgrades for big jumps.

Code & diagrams

node maintenancebash

kubectl cordon ip-10-20-11-14.ap-south-1.compute.internal
kubectl drain ip-10-20-11-14.ap-south-1.compute.internal --ignore-daemonsets --delete-emptydir-data --timeout=10m
# ...patch / replace / upgrade the node...
kubectl uncordon ip-10-20-11-14.ap-south-1.compute.internal
kubectl get nodes -o wide

etcd backup and restore (kubeadm)bash

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd-$(date +%F-%H%M).db
etcdutl snapshot status /backup/etcd-2026-09-28-0200.db --write-out=table
# restore (control plane stopped): creates a new data dir, then point etcd at it
etcdutl snapshot restore /backup/etcd-2026-09-28-0200.db --data-dir /var/lib/etcd-restore

upgrade orderdiagram

Rendering diagram…

Explain it without notes

Why upgrade one minor version at a time?

Practice

Write a runbook for upgrading a 3-node self-managed cluster from 1.33 to 1.34.

Trade-offs

↔
In-place upgrades are cheaper but riskier; blue-green clusters (build new, shift traffic via GitOps and DNS) cost more but make rollback trivial.

Done when you can

I can safely cordon, drain, and uncordon nodes
I can plan a version upgrade and back up/restore etcd