Command Palette

Search for a command to run...

Hectal

Guide G6 · DevOps path

The Advanced Kubernetes Ecosystem

A map of the add-ons production clusters run: CNI and eBPF networking, CSI storage, service mesh (mTLS, retries, traffic shifting), KEDA event-driven autoscaling, Falco runtime security, and safe cluster upgrades.

Advanced 45 min

Start here

The mental model

Kubernetes itself is a small core: an API, a scheduler, and controllers. Almost everything a production cluster does (networking, storage, certificates, DNS, autoscaling, security, delivery) comes from PLUG-INS and ADD-ONS that implement standard interfaces or extend the API with CRDs. Knowing the map (which box does what, and which interface it plugs into) lets you read any company's cluster in minutes.

Go deeper

How it works inside

01The plug-in interfaces: CRI, CNI, CSI

CRI (Container Runtime Interface): how the kubelet talks to containerd or CRI-O (Docker course, runtime stack). CNI (Container Network Interface): gives each pod an IP and connectivity. The AWS VPC CNI uses real VPC IPs, Calico adds network policy and BGP, and CILIUM uses eBPF for fast routing, network policy (including L7), load balancing without kube-proxy, and observability with Hubble (Networking course, Kubernetes networking). CSI (Container Storage Interface): how volumes are provisioned and attached: the EBS CSI driver, EFS CSI, and others, including snapshots via VolumeSnapshot (Stateful Systems course, Unit 0.2).

02Service mesh

A SERVICE MESH moves service-to-service concerns out of application code into the infrastructure: automatic mutual TLS (every call encrypted and authenticated by workload identity), retries, timeouts, and circuit breaking, fine-grained traffic shifting for canaries, and uniform golden-signal metrics and traces for every call. ISTIO is the most featureful (its sidecar-less AMBIENT mode uses a per-node proxy for L4 and optional waypoint proxies for L7, cutting overhead); LINKERD is simpler and lightweight; Cilium offers mesh features with eBPF. Adopt one when you have many services and need zero-trust networking or consistent traffic control, not by default (Kubernetes course, service mesh basics).

03Autoscaling beyond CPU: KEDA

HPA scales on CPU/memory or custom metrics. KEDA (Kubernetes Event-Driven Autoscaling) adds 60+ SCALERS (Kafka consumer lag, SQS queue depth, RabbitMQ, Prometheus queries, cron schedules) and can scale to ZERO when there's no work. A Kafka consumer that scales with lag, or a worker that only runs while a queue has messages, is the classic use (Stateful Systems course, consumer lag).

04Runtime security: Falco

Admission policies (Kyverno, Gatekeeper) check what's DEPLOYED; FALCO (CNCF) watches what containers DO at runtime using eBPF: a shell spawned in a production container, a write to /etc, an unexpected outbound connection, reading sensitive files. Alerts go to Slack/SIEM via Falcosidekick, and can trigger response automation (DevSecOps course, runtime detection).

05Cluster upgrades

Kubernetes releases a minor version about every four months, and managed services support each for a limited window (EKS standard support ~14 months, then paid extended support). Upgrade one minor version at a time: check deprecated APIs first (kubent, pluto, EKS upgrade insights), upgrade add-ons to compatible versions, upgrade the control plane, then node groups (or let Karpenter's drift detection replace nodes with the new AMI). Rehearse in staging, keep PDBs correct so drains don't cause outages, and consider blue-green CLUSTER upgrades (new cluster, move traffic via GitOps + DNS) for large version jumps.

Cluster upgradesdiagram
Rendering diagram…

Do it

Hands-on lab

  1. 1

    Scale a Kafka consumer on lag with KEDA

    The ScaledObject targets the email consumer Deployment: one replica per 1,000 messages of lag, up to the partition count, and zero when idle.

    apps/email-consumer/base/scaledobject.yamlwhole fileyaml
    apiVersion: keda.sh/v1alpha1
    kind: ScaledObject
    metadata:
      name: email-consumer
    spec:
      scaleTargetRef: { name: email-consumer }
      minReplicaCount: 0
      maxReplicaCount: 12          # = partitions of the topic
      cooldownPeriod: 300
      triggers:
        - type: kafka
          metadata:
            bootstrapServers: shop-kafka-bootstrap.kafka:9092
            consumerGroup: email
            topic: payments
            lagThreshold: "1000"
    terminal
    $ kubectl get scaledobject email-consumer; kubectl get hpa keda-hpa-email-consumer
    ── expected output ──
    NAME SCALETARGETKIND SCALETARGETNAME MIN MAX READY ACTIVE
    email-consumer apps/v1.Deployment email-consumer 0 12 True True
    NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS
    keda-hpa-email-consumer Deployment/email-consumer 4200/1k (avg) 1 12 5
  2. 2

    Check for deprecated APIs before an upgrade

    pluto scans live clusters and Helm releases for APIs removed in the target version.

    terminal
    $ pluto detect-all-in-cluster --target-versions k8s=v1.34.0 -o wide
    ── expected output ──
    NAME NAMESPACE KIND VERSION REPLACEMENT DEPRECATED REMOVED
    shop-hpa catalog HorizontalPodAutoscaler autoscaling/v2beta2 autoscaling/v2 true true
  3. 3

    A Falco rule for shells in production

    Falco ships a similar default rule; this custom version scopes it to ShopLite's prod namespaces and raises priority.

    falco/rules/shoplite.yamlwhole fileyaml
    - rule: Shell in ShopLite production container
      desc: A shell was started inside a production workload container
      condition: >
        spawned_process and container and proc.name in (bash, sh, zsh)
        and k8s.ns.name startswith "shoplite-prod"
      output: >
        Shell in prod (user=%user.name pod=%k8s.pod.name ns=%k8s.ns.name
        cmd=%proc.cmdline image=%container.image.repository)
      priority: CRITICAL
      tags: [shoplite, mitre_execution]

Operate it

Knobs that matter

SettingDefaultWhat it doesWhen to change it
EKS version support window~14 months standardHow long a minor version is supported without extra cost.Plan upgrades at least twice a year; never fall into extended support by accident.
KEDA cooldownPeriod300sWait before scaling back to zero.Longer for bursty queues to avoid cold starts.
Mesh mTLS modepermissive (Istio)Accept plain-text and mTLS during migration.Switch to STRICT once all workloads are in the mesh.

3am practice

Failure drills

Each drill is a real failure mode. Read the scenario and the output, decide what's wrong, then reveal the diagnosis.

Drill #1

Upgrade removes an API a Helm release still uses

The cluster is upgraded to a version where an old API (e.g. autoscaling/v2beta2) was removed. A later helm upgrade of the catalog chart fails.

terminal
$ helm upgrade catalog charts/catalog -n catalog
── what you'll see ──
Error: UPGRADE FAILED: unable to build kubernetes objects from current release manifest: resource mapping not found for name: "shop-hpa" namespace: "" from "": no matches for kind "HorizontalPodAutoscaler" in version "autoscaling/v2beta2"
ensure CRDs are installed first

Decide

Which add-on solves which problem

ProblemAdd-onWhere it's covered
Pod networking and network policyVPC CNI, Calico, CiliumNetworking course, phase 6
Persistent volumes, snapshotsEBS/EFS CSI driversStateful Systems, Unit 0.2
Ingress / routingGateway API implementationsPlatform course, Mission 0.2
Certificates and DNScert-manager, ExternalDNSPlatform course, Mission 0.2
Secrets from a vaultExternal Secrets OperatorGitOps course, Mission 2.1
Node autoscalingKarpenter, Cluster AutoscalerPlatform course, Mission 0.3
Event-driven autoscalingKEDAThis guide
Admission policyKyverno, GatekeeperPlatform course, Mission 0.4
Runtime threat detectionFalcoThis guide + DevSecOps course
mTLS, retries, traffic shiftingIstio, Linkerd, Cilium meshThis guide + Kubernetes phase 8
DeliveryArgo CD, Flux, Argo RolloutsGitOps course
Cloud resources as CRDsCrossplanePlatform course, Stage 2

The bigger picture

Connects to

Prove it

Interview questions

01

What are CNI and CSI?

02

When would you introduce a service mesh?

03

How do you upgrade a Kubernetes cluster safely?