Command Palette

Search for a command to run...

Hectal

Mission 1.3 · Stage 1 — GitOps with Argo CD

Automated Sync, Self-Heal, Pruning, and Sync Waves

Goal: Argo CD deploys every merge automatically, reverts manual cluster edits, deletes what's removed from Git, and applies resources in the right order.

40 min Free 5 steps 2 break-it drills

By the end of this mission

  • Turn on automated sync with prune and selfHeal, and know the risks of each
  • Watch Argo detect and undo drift
  • Ignore fields that other controllers legitimately own (HPA replicas)
  • Order resources with sync waves and run migrations with hooks

Part 1

Understand it first

Three switches, three behaviours

automated: {}: sync whenever Git changes. prune: true: delete live resources that were removed from Git (off by default, because deleting is the dangerous direction). selfHeal: true: when the LIVE state changes (someone ran kubectl edit), put it back to match Git, usually within seconds.

With all three on, Git is the only way to change the cluster. That's the goal, but it has consequences: an on-call engineer's emergency kubectl scale will be reverted. The emergency path must be 'commit to Git' (fast-tracked review) or, in rare cases, temporarily disabling auto-sync on that app. Write that into your runbook (SRE course, incident response).

Drift and ignoreDifferences

DRIFT is any difference between live and desired state. Some differences are legitimate: an HPA changes spec.replicas, a mutating webhook injects a sidecar, a controller adds default fields. ignoreDifferences tells Argo to leave specific fields alone; otherwise it fights the HPA forever. Simplest rule for HPAs: don't set replicas in Git at all.

Sync waves and hooks

By default Argo applies resources in a sensible kind order (Namespaces, then ConfigMaps/Secrets, then Deployments). SYNC WAVES (argocd.argoproj.io/sync-wave: "-1") give explicit ordering: lower waves apply first, and each wave must be Healthy before the next starts. HOOKS are resources (usually Jobs) that run at a phase: PreSync (database migrations), Sync, PostSync (smoke tests, notifications), SyncFail (cleanup). A failing PreSync migration stops the rollout before new code runs against an old schema.

One sync with waves and hooksdiagram
Rendering diagram…

Part 2

Your project after this mission · 4 files change

shoplite-gitops/
  • shoplite-gitops/
    • apps/
      • web/
        • deployment.yamlmodified
        • hpa.yamlnew
        • migrate-job.yamlnew
    • argocd/
      • web-dev.yamlmodified

Part 3

Build it, step by step

  1. 1

    Turn on automation

    retry handles transient failures (a CRD not yet ready). revisionHistoryLimit keeps the history list short.

    shoplite-gitops/argocd/web-dev.yamlwhole fileyaml
    apiVersion: argoproj.io/v1alpha1
    kind: Application
    metadata:
      name: web-dev
      namespace: argocd
    spec:
      project: default
      source:
        repoURL: https://github.com/<you>/shoplite-gitops.git
        targetRevision: main
        path: apps/web
      destination:
        server: https://kubernetes.default.svc
        namespace: shoplite-dev
      revisionHistoryLimit: 10
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        retry:
          limit: 3
          backoff: { duration: 10s, factor: 2, maxDuration: 2m }
        syncOptions:
          - CreateNamespace=true
      ignoreDifferences:
        - group: apps
          kind: Deployment
          jsonPointers: [/spec/replicas]
  2. 2

    Watch self-heal undo a manual change

    Scale by hand and watch. Within seconds Argo notices live ≠ Git and restores it. Here the replicas field is ignored, so try an image change instead, which is exactly the kind of 'quick fix' GitOps forbids.

    terminal
    $ kubectl apply -f argocd/web-dev.yaml
    kubectl -n shoplite-dev set image deploy/web web=nginxdemos/hello:latest
    sleep 10; kubectl -n shoplite-dev get deploy web -o jsonpath='{.spec.template.spec.containers[0].image}'
    argocd app history web-dev | tail -1
    ── expected output ──
    deployment.apps/web image updated
    nginxdemos/hello:0.3
    2 2026-09-27 10:31:02 +0530 IST main (d4e5f6a)
  3. 3

    Let the HPA own replicas

    Remove replicas: 2 from the Deployment and add an HPA (Kubernetes course, autoscaling). With the ignoreDifferences entry, Argo stays Synced while the HPA scales.

    shoplite-gitops/apps/web/hpa.yamlwhole fileyaml
    apiVersion: autoscaling/v2
    kind: HorizontalPodAutoscaler
    metadata:
      name: web
    spec:
      scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: web }
      minReplicas: 2
      maxReplicas: 10
      metrics:
        - type: Resource
          resource: { name: cpu, target: { type: Utilization, averageUtilization: 70 } }
  4. 4

    Run a migration before the new version starts

    A PreSync hook Job runs before anything else in the sync. hook-delete-policy: BeforeHookCreation deletes the previous run's Job so the name can be reused. Migrations must be backward compatible (expand, then contract), because old pods keep running during the rollout.

    shoplite-gitops/apps/web/migrate-job.yamlwhole fileyaml
    apiVersion: batch/v1
    kind: Job
    metadata:
      name: db-migrate
      annotations:
        argocd.argoproj.io/hook: PreSync
        argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
    spec:
      backoffLimit: 1
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: migrate
              image: busybox:1.37
              command: ["sh", "-c", "echo running migrations for 0.3 && sleep 5"]
    terminal
    $ git add . && git commit -m 'web: hpa + presync migration' && git push
    argocd app wait web-dev --sync --health && argocd app get web-dev | grep -A6 HOOK
    ── expected output ──
    GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE
    batch Job shoplite-dev db-migrate Succeeded PreSync job.batch/db-migrate created
    ...
  5. 5

    Prune: remove from Git, removed from cluster

    Delete the HPA file, push, and it disappears from the cluster. Without prune: true it would stay forever as an orphan, marked OutOfSync with a 'requires pruning' flag.

    terminal
    $ git rm apps/web/hpa.yaml && git commit -m 'web: drop hpa' && git push
    argocd app wait web-dev --sync && kubectl -n shoplite-dev get hpa
    ── expected output ──
    No resources found in shoplite-dev namespace.

Checkpoint — you should now have

  • ✓A manual kubectl set image is reverted within seconds.
  • ✓The PreSync migration Job runs before each sync.
  • ✓Removing a file from Git removes the resource from the cluster.

Part 4

Break it on purpose

Make each change, run the command, and read the error before revealing the diagnosis. Recognising these messages on sight is what makes you fast on a real team. Undo the change afterwards.

Break #1

Fight the autoscaler

Put replicas: 2 back in the Deployment and remove the ignoreDifferences block, with the HPA present and some CPU load.

terminal
$ kubectl -n shoplite-dev get deploy web -w
── what you'll see ──
NAME READY UP-TO-DATE AVAILABLE
web 5/5 5 5
web 2/2 2 2
web 2/5 5 2
web 2/2 2 2

Break #2

A failing migration

Change the migrate command to exit 1 and push.

terminal
$ argocd app get web-dev
── what you'll see ──
Sync Status: OutOfSync from main (b7c8d9e)
Health Status: Healthy
Operation: Sync
Phase: Failed
Message: one or more synchronization tasks completed unsuccessfully (job db-migrate: Job has reached the specified backoff limit)

Part 5

Interview questions from this mission

01

What do prune and selfHeal do, and why are they off by default?

02

How do you run database migrations in a GitOps workflow?

0/4 · 0%