Topic 4.2
Affinity, Taints & Tolerations
In one line
Affinity rules pull pods toward (or away from) specific nodes or other pods; taints and tolerations do the opposite — repelling pods from a node unless they're specifically permitted to land there.
Think of it like this
Affinity is like a personal preference ('I'd like to sit near the window, or near my colleague') — a pull TOWARD something. A taint is like a 'staff only' sign on a door — it repels everyone by default, and only someone with the specific right badge (a matching toleration) is allowed through anyway.
Key ideas
- 01
NODE AFFINITY constrains which nodes a pod can be scheduled onto, based on the NODE's own labels —
requiredDuringSchedulingIgnoredDuringExecutionis a HARD requirement (the pod won't schedule at all without a matching node), whilepreferredDuringSchedulingIgnoredDuringExecutionis a SOFT preference (the scheduler tries to honor it, but will schedule elsewhere if it genuinely can't). - 02
POD AFFINITY and POD ANTI-AFFINITY constrain placement based on OTHER PODS already running, rather than node labels directly — pod affinity pulls a pod TOWARD nodes already running certain other pods (useful for co-locating pods that communicate heavily, reducing network latency between them); pod ANTI-affinity pushes pods AWAY from nodes already running certain other pods (the standard way to spread a Deployment's replicas across DIFFERENT nodes, so a single node failure can't take out every copy at once).
- 03
A TAINT is applied to a NODE, marking it as generally undesirable or restricted (
kubectl taint nodes node1 dedicated=gpu:NoSchedule) — by default, NO pod will be scheduled onto a tainted node at all. A TOLERATION is applied to a POD, explicitly stating it can tolerate (ignore) a specific taint — only pods with a matching toleration will be considered for scheduling onto that tainted node. - 04
Taints and tolerations solve a fundamentally different problem than affinity: affinity is a pod EXPRESSING A PREFERENCE for certain nodes; taints/tolerations are a NODE actively REPELLING pods unless they're specifically permitted. The classic real use case: dedicating expensive GPU nodes exclusively to workloads that actually need a GPU, by tainting those nodes and only giving GPU-requiring pods the matching toleration — everything else is automatically kept off them.
- 05
A common, genuinely important REAL pattern combines these: pod anti-affinity to spread a Deployment's replicas across different AVAILABILITY ZONES (not just different nodes), so an entire zone outage can't take down every replica simultaneously — directly connecting to Linux's own course on high availability concepts, now expressed as native Kubernetes scheduling rules.
Code & diagrams
A tainted node repels every pod by default — only a matching toleration gets through.
Anti-affinity spreading replicas across zones, plus a toleration for a tainted GPU node.
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 3
selector:
matchLabels: { app: my-app }
template:
metadata:
labels: { app: my-app }
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels: { app: my-app }
topologyKey: "topology.kubernetes.io/zone" # spread across zones
containers:
- name: my-app
image: my-registry/my-app:1.5.0
---
apiVersion: v1
kind: Pod
metadata:
name: gpu-training-job
spec:
tolerations:
- key: "dedicated"
operator: "Equal"
value: "gpu"
effect: "NoSchedule" # matches the node's taint exactly
containers:
- name: trainer
image: my-registry/gpu-trainer:1.0.0Apply a taint, confirm the repelling behavior, then confirm the toleration overrides it.
# Taint a node — nothing will schedule here by default afterward
kubectl taint nodes node-1 dedicated=gpu:NoSchedule
# A normal pod (no toleration) will simply never land on node-1
kubectl get pods -o wide
# Only a pod with the matching toleration can be scheduled there
kubectl apply -f affinity-and-taints.yaml
kubectl get pod gpu-training-job -o wide
# NODE column should show node-1
# Remove the taint if needed
kubectl taint nodes node-1 dedicated=gpu:NoSchedule-Explain it without notes
What's the fundamental difference between what affinity does and what a taint does, in terms of who's expressing the preference?
Why would a team specifically taint their cluster's GPU nodes rather than just relying on regular pods simply not requesting a GPU?
Practice
Taint a test node, confirm no ordinary pod gets scheduled onto it, then create a pod with a matching toleration and confirm it does land there.
Add pod anti-affinity to a Deployment (matching this topic's zone-spreading example) and, on a multi-zone cluster, confirm with kubectl get pods -o wide that replicas actually land on different zones/nodes.
Trade-offs
- ↔
Hard requirements (
required...affinity, or a taint with no matching toleration anywhere) give strong, predictable guarantees but can leave pods permanentlyPendingif the exact conditions can never be met — soft preferences (preferred...) are more flexible and won't block scheduling entirely, but offer no hard guarantee the preference is actually honored, which is exactly the trade-off to weigh based on how critical a given placement constraint genuinely is.
Done when you can
I can explain the difference between node affinity, pod affinity, and pod anti-affinity.
I understand taints repel pods by default, and only a matching toleration allows scheduling anyway.
I can use pod anti-affinity to spread a Deployment's replicas across nodes or zones.