Guide G6 · DevOps path
The Advanced Kubernetes Ecosystem
A map of the add-ons production clusters run: CNI and eBPF networking, CSI storage, service mesh (mTLS, retries, traffic shifting), KEDA event-driven autoscaling, Falco runtime security, and safe cluster upgrades.
Start here
The mental model
Kubernetes itself is a small core: an API, a scheduler, and controllers. Almost everything a production cluster does (networking, storage, certificates, DNS, autoscaling, security, delivery) comes from PLUG-INS and ADD-ONS that implement standard interfaces or extend the API with CRDs. Knowing the map (which box does what, and which interface it plugs into) lets you read any company's cluster in minutes.
Go deeper
How it works inside
01The plug-in interfaces: CRI, CNI, CSI
CRI (Container Runtime Interface): how the kubelet talks to containerd or CRI-O (Docker course, runtime stack). CNI (Container Network Interface): gives each pod an IP and connectivity. The AWS VPC CNI uses real VPC IPs, Calico adds network policy and BGP, and CILIUM uses eBPF for fast routing, network policy (including L7), load balancing without kube-proxy, and observability with Hubble (Networking course, Kubernetes networking). CSI (Container Storage Interface): how volumes are provisioned and attached: the EBS CSI driver, EFS CSI, and others, including snapshots via VolumeSnapshot (Stateful Systems course, Unit 0.2).
02Service mesh
A SERVICE MESH moves service-to-service concerns out of application code into the infrastructure: automatic mutual TLS (every call encrypted and authenticated by workload identity), retries, timeouts, and circuit breaking, fine-grained traffic shifting for canaries, and uniform golden-signal metrics and traces for every call. ISTIO is the most featureful (its sidecar-less AMBIENT mode uses a per-node proxy for L4 and optional waypoint proxies for L7, cutting overhead); LINKERD is simpler and lightweight; Cilium offers mesh features with eBPF. Adopt one when you have many services and need zero-trust networking or consistent traffic control, not by default (Kubernetes course, service mesh basics).
03Autoscaling beyond CPU: KEDA
HPA scales on CPU/memory or custom metrics. KEDA (Kubernetes Event-Driven Autoscaling) adds 60+ SCALERS (Kafka consumer lag, SQS queue depth, RabbitMQ, Prometheus queries, cron schedules) and can scale to ZERO when there's no work. A Kafka consumer that scales with lag, or a worker that only runs while a queue has messages, is the classic use (Stateful Systems course, consumer lag).
04Runtime security: Falco
Admission policies (Kyverno, Gatekeeper) check what's DEPLOYED; FALCO (CNCF) watches what containers DO at runtime using eBPF: a shell spawned in a production container, a write to /etc, an unexpected outbound connection, reading sensitive files. Alerts go to Slack/SIEM via Falcosidekick, and can trigger response automation (DevSecOps course, runtime detection).
05Cluster upgrades
Kubernetes releases a minor version about every four months, and managed services support each for a limited window (EKS standard support ~14 months, then paid extended support). Upgrade one minor version at a time: check deprecated APIs first (kubent, pluto, EKS upgrade insights), upgrade add-ons to compatible versions, upgrade the control plane, then node groups (or let Karpenter's drift detection replace nodes with the new AMI). Rehearse in staging, keep PDBs correct so drains don't cause outages, and consider blue-green CLUSTER upgrades (new cluster, move traffic via GitOps + DNS) for large version jumps.
Do it
Hands-on lab
- 1
Scale a Kafka consumer on lag with KEDA
The ScaledObject targets the email consumer Deployment: one replica per 1,000 messages of lag, up to the partition count, and zero when idle.
apps/email-consumer/base/scaledobject.yamlwhole fileyaml apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: email-consumer spec: scaleTargetRef: { name: email-consumer } minReplicaCount: 0 maxReplicaCount: 12 # = partitions of the topic cooldownPeriod: 300 triggers: - type: kafka metadata: bootstrapServers: shop-kafka-bootstrap.kafka:9092 consumerGroup: email topic: payments lagThreshold: "1000"terminal$ kubectl get scaledobject email-consumer; kubectl get hpa keda-hpa-email-consumer── expected output ──NAME SCALETARGETKIND SCALETARGETNAME MIN MAX READY ACTIVEemail-consumer apps/v1.Deployment email-consumer 0 12 True TrueNAME REFERENCE TARGETS MINPODS MAXPODS REPLICASkeda-hpa-email-consumer Deployment/email-consumer 4200/1k (avg) 1 12 5 - 2
Check for deprecated APIs before an upgrade
plutoscans live clusters and Helm releases for APIs removed in the target version.terminal$ pluto detect-all-in-cluster --target-versions k8s=v1.34.0 -o wide── expected output ──NAME NAMESPACE KIND VERSION REPLACEMENT DEPRECATED REMOVEDshop-hpa catalog HorizontalPodAutoscaler autoscaling/v2beta2 autoscaling/v2 true true - 3
A Falco rule for shells in production
Falco ships a similar default rule; this custom version scopes it to ShopLite's prod namespaces and raises priority.
falco/rules/shoplite.yamlwhole fileyaml - rule: Shell in ShopLite production container desc: A shell was started inside a production workload container condition: > spawned_process and container and proc.name in (bash, sh, zsh) and k8s.ns.name startswith "shoplite-prod" output: > Shell in prod (user=%user.name pod=%k8s.pod.name ns=%k8s.ns.name cmd=%proc.cmdline image=%container.image.repository) priority: CRITICAL tags: [shoplite, mitre_execution]
Operate it
Knobs that matter
| Setting | Default | What it does | When to change it |
|---|---|---|---|
| EKS version support window | ~14 months standard | How long a minor version is supported without extra cost. | Plan upgrades at least twice a year; never fall into extended support by accident. |
| KEDA cooldownPeriod | 300s | Wait before scaling back to zero. | Longer for bursty queues to avoid cold starts. |
| Mesh mTLS mode | permissive (Istio) | Accept plain-text and mTLS during migration. | Switch to STRICT once all workloads are in the mesh. |
3am practice
Failure drills
Each drill is a real failure mode. Read the scenario and the output, decide what's wrong, then reveal the diagnosis.
Drill #1
Upgrade removes an API a Helm release still uses
The cluster is upgraded to a version where an old API (e.g. autoscaling/v2beta2) was removed. A later helm upgrade of the catalog chart fails.
Decide
Which add-on solves which problem
| Problem | Add-on | Where it's covered |
|---|---|---|
| Pod networking and network policy | VPC CNI, Calico, Cilium | Networking course, phase 6 |
| Persistent volumes, snapshots | EBS/EFS CSI drivers | Stateful Systems, Unit 0.2 |
| Ingress / routing | Gateway API implementations | Platform course, Mission 0.2 |
| Certificates and DNS | cert-manager, ExternalDNS | Platform course, Mission 0.2 |
| Secrets from a vault | External Secrets Operator | GitOps course, Mission 2.1 |
| Node autoscaling | Karpenter, Cluster Autoscaler | Platform course, Mission 0.3 |
| Event-driven autoscaling | KEDA | This guide |
| Admission policy | Kyverno, Gatekeeper | Platform course, Mission 0.4 |
| Runtime threat detection | Falco | This guide + DevSecOps course |
| mTLS, retries, traffic shifting | Istio, Linkerd, Cilium mesh | This guide + Kubernetes phase 8 |
| Delivery | Argo CD, Flux, Argo Rollouts | GitOps course |
| Cloud resources as CRDs | Crossplane | Platform course, Stage 2 |
The bigger picture
Connects to
System Design · Service Discovery
Finding a service instance by NAME, not IP: registry + heartbeats, or DNS-based resolution.
Networking · Kubernetes networking, CNI & eBPF
The CNI layer in depth.
Kubernetes · Service mesh basics
The fundamentals of sidecars and mesh traffic.
Kubernetes · Cluster Autoscaler & KEDA
Where autoscaling was introduced.
Stateful · Consumer lag
The metric KEDA scales Kafka consumers on.
Prove it
Interview questions
What are CNI and CSI?
When would you introduce a service mesh?
How do you upgrade a Kubernetes cluster safely?