Capacity on Demand: Karpenter and Cost-Aware Nodes

1. 1

   Prerequisites from Terraform

   Karpenter needs an IAM role for its controller (Pod Identity), a node IAM role, an SQS interruption queue with EventBridge rules, and subnets and security groups tagged karpenter.sh/discovery: shoplite-prod. The official Terraform module (terraform-aws-modules/eks/aws//modules/karpenter) creates all of it. Add it to the EKS stack from the AWS/Terraform courses and apply.

   terminalcopy cmd

   $ terraform -chdir=envs/prod apply -target=module.karpenter

   aws sqs list-queues --queue-name-prefix Karpenter

   ── expected output ──

   Apply complete! Resources: 11 added, 0 changed, 0 destroyed.

   { "QueueUrls": [ "https://sqs.ap-south-1.amazonaws.com/123456789012/Karpenter-shoplite-prod" ] }

2. 2

   Install the controller via Argo CD

   Chart oci://public.ecr.aws/karpenter/karpenter. Run the controller itself on a small fixed managed node group (not on nodes it manages).

   apiVersion: argoproj.io/v1alpha1
   kind: Application
   metadata:
     name: karpenter
     namespace: argocd
   spec:
     project: default
     source:
       repoURL: public.ecr.aws/karpenter
       chart: karpenter
       targetRevision: 1.7.1
       helm:
         valuesObject:
           settings:
             clusterName: shoplite-prod
             interruptionQueue: Karpenter-shoplite-prod
           nodeSelector: { role: system }
     destination:
       server: https://kubernetes.default.svc
       namespace: kube-system
     syncPolicy:
       automated: { prune: true, selfHeal: true }

3. 3

   Define the node pools

   general allows Spot and on-demand across many instance families (flexibility is what makes Spot reliable) with a CPU limit as a cost guardrail. consolidationPolicy: WhenEmptyOrUnderutilized lets Karpenter repack pods. budgets limit how many nodes can be disrupted at once. A second on-demand pool with a taint is for stateful workloads that tolerate it.

   apiVersion: karpenter.k8s.aws/v1
   kind: EC2NodeClass
   metadata: { name: default }
   spec:
     amiSelectorTerms: [{ alias: al2023@latest }]
     role: KarpenterNodeRole-shoplite-prod
     subnetSelectorTerms: [{ tags: { karpenter.sh/discovery: shoplite-prod } }]
     securityGroupSelectorTerms: [{ tags: { karpenter.sh/discovery: shoplite-prod } }]
     blockDeviceMappings:
       - deviceName: /dev/xvda
         ebs: { volumeSize: 50Gi, volumeType: gp3, encrypted: true }
   ---
   apiVersion: karpenter.sh/v1
   kind: NodePool
   metadata: { name: general }
   spec:
     template:
       spec:
         nodeClassRef: { group: karpenter.k8s.aws, kind: EC2NodeClass, name: default }
         requirements:
           - { key: karpenter.sh/capacity-type, operator: In, values: [spot, on-demand] }
           - { key: kubernetes.io/arch, operator: In, values: [amd64, arm64] }
           - { key: karpenter.k8s.aws/instance-category, operator: In, values: [c, m, r] }
           - { key: karpenter.k8s.aws/instance-generation, operator: Gt, values: ["5"] }
         expireAfter: 720h          # recycle nodes monthly (fresh AMIs, patches)
     limits: { cpu: "200" }
     disruption:
       consolidationPolicy: WhenEmptyOrUnderutilized
       consolidateAfter: 1m
       budgets: [{ nodes: "10%" }]
   ---
   apiVersion: karpenter.sh/v1
   kind: NodePool
   metadata: { name: on-demand-stateful }
   spec:
     template:
       spec:
         nodeClassRef: { group: karpenter.k8s.aws, kind: EC2NodeClass, name: default }
         requirements:
           - { key: karpenter.sh/capacity-type, operator: In, values: [on-demand] }
         taints: [{ key: shoplite.dev/stateful, effect: NoSchedule }]
     disruption: { consolidationPolicy: WhenEmpty, consolidateAfter: 10m }

4. 4

   Watch it provision and consolidate

   Scale a test deployment so pods go Pending, and watch a NodeClaim appear and become a Ready node. Scale down and watch consolidation remove it.

   terminalcopy cmd

   $ kubectl create deploy inflate --image=public.ecr.aws/eks-distro/kubernetes/pause:3.10 --replicas=0

   kubectl set resources deploy inflate --requests=cpu=1

   kubectl scale deploy inflate --replicas=20

   kubectl get nodeclaims -w

   ── expected output ──

   NAME TYPE CAPACITY ZONE NODE READY

   general-8x2kq c7g.4xlarge spot ap-south-1b Unknown

   general-8x2kq c7g.4xlarge spot ap-south-1b ip-10-20-12-87.ap-south-1.compute.internal True

5. 5

   Scale down and see consolidation

   After consolidateAfter, Karpenter terminates the empty node. kubectl get events shows its reasoning.

   terminalcopy cmd

   $ kubectl scale deploy inflate --replicas=0

   kubectl get events -A --field-selector source=karpenter --sort-by=.lastTimestamp | tail -2

   ── expected output ──

   Normal DisruptionTerminating nodeclaim/general-8x2kq Disrupting NodeClaim: Empty/Delete

   Normal Finalized node/ip-10-20-12-87... Finalized karpenter.sh/termination

Make each change, run the command, and read the error before revealing the diagnosis. Recognising these messages on sight is what makes you fast on a real team. Undo the change afterwards.