Command Palette

Search for a command to run...

Hectal

Mission 1.4 · Stage 1 — GitOps with Argo CD

Environments, Overlays, and the CI → Git Handoff

Goal: Dev and prod from one base using Kustomize overlays, a Helm-sourced dependency, CI that bumps image tags by commit, and an app-of-apps root.

60 min Free 9 steps 2 break-it drills

By the end of this mission

  • Structure a config repo with a base and per-environment overlays
  • Deploy a Helm chart through Argo CD with values in Git
  • Have CI update the image tag in the config repo instead of deploying
  • Promote dev → prod with a pull request
  • Bootstrap everything from one root Application (app of apps) and restrict it with an AppProject

Part 1

Understand it first

Base and overlays

Copy-pasting manifests per environment drifts quickly. KUSTOMIZE keeps one base/ and small overlays/<env>/ that patch only what differs (replicas, resources, image tag, hostnames). HELM does the same with one chart and a values file per environment (Kubernetes course, Helm topic). Both work in Argo CD. Common choice: Helm for third-party software (you consume someone's chart), Kustomize for your own apps. You can also combine them.

Environments as FOLDERS on one branch (overlays/dev, overlays/prod) are the recommended layout. Environments as BRANCHES (dev, prod branches) sound natural but make promotion a merge between long-lived branches full of conflicts and unrelated changes.

The CI → Git handoff

The app repo's pipeline (CI/CD course) still builds, tests, scans, and pushes the image, tagged with the Git SHA, never latest. Its last step changes: instead of kubectl apply, it commits the new tag into the config repo's dev overlay (kustomize edit set image). Argo deploys it. Promotion to prod is a PR that copies the same tag into the prod overlay, so the image that was tested in dev is byte-for-byte the one that ships.

Argo CD Image Updater can automate the dev bump by watching the registry, but an explicit CI commit is simpler to reason about and to audit.

App of apps and AppProjects

Applying Application YAML by hand doesn't scale. In the APP OF APPS pattern, one root Application points at the argocd/ folder of Applications, so adding a new app is just adding a file. A cluster can be rebuilt by installing Argo CD and applying one root Application.

An APPPROJECT fences Applications: which repos they may use, which clusters/namespaces they may deploy to, and which kinds they may create (for example, no ClusterRoles for app teams). It's Argo CD's multi-tenancy boundary, used with its RBAC and SSO groups.

From commit to proddiagram
Rendering diagram…

Part 2

Your project after this mission · 11 files change

shoplite-gitops/
  • shoplite-gitops/
    • apps/
      • web/
        • base/
          • deployment.yamlnew
          • kustomization.yamlnew
          • service.yamlnew
        • overlays/
          • dev/
            • kustomization.yamlnew
          • prod/
            • kustomization.yamlnew
    • argocd/
      • apps/
        • redis-dev.yamlnew
        • web-dev.yamlmodified
        • web-prod.yamlnew
      • projects/
        • shoplite.yamlnew
      • root.yamlnew
  • shoplite-web/
    • .github/
      • workflows/
        • release.ymlmodified

Part 3

Build it, step by step

  1. 1

    Move manifests into a base

    Move deployment.yaml, service.yaml, and the migrate Job into apps/web/base/ and list them in a kustomization.

    shoplite-gitops/apps/web/base/kustomization.yamlwhole fileyaml
    apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    resources:
      - deployment.yaml
      - service.yaml
      - migrate-job.yaml
    labels:
      - pairs: { app.kubernetes.io/part-of: shoplite }
  2. 2

    The dev overlay

    The images block is what CI will edit. newTag pins an immutable tag.

    shoplite-gitops/apps/web/overlays/dev/kustomization.yamlwhole fileyaml
    apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    namespace: shoplite-dev
    resources: [../../base]
    images:
      - name: nginxdemos/hello
        newTag: "0.4"
    patches:
      - target: { kind: Deployment, name: web }
        patch: |
          - op: add
            path: /spec/replicas
            value: 1
  3. 3

    The prod overlay

    More replicas, bigger requests, a PodDisruptionBudget would go here too. Prod lags dev by one version until it's promoted.

    shoplite-gitops/apps/web/overlays/prod/kustomization.yamlwhole fileyaml
    apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    namespace: shoplite-prod
    resources: [../../base]
    images:
      - name: nginxdemos/hello
        newTag: "0.3"
    patches:
      - target: { kind: Deployment, name: web }
        patch: |
          - op: add
            path: /spec/replicas
            value: 3
          - op: replace
            path: /spec/template/spec/containers/0/resources/requests/cpu
            value: 250m
    terminal
    $ kustomize build apps/web/overlays/prod | grep -E 'namespace|replicas|image:|cpu'
    ── expected output ──
    namespace: shoplite-prod
    replicas: 3
    image: nginxdemos/hello:0.3
    cpu: 250m
  4. 4

    Fence it with an AppProject

    ShopLite apps may only come from this repo and the Bitnami chart registry, only deploy to shoplite-* namespaces, and can't create cluster-scoped resources except Namespaces.

    shoplite-gitops/argocd/projects/shoplite.yamlwhole fileyaml
    apiVersion: argoproj.io/v1alpha1
    kind: AppProject
    metadata:
      name: shoplite
      namespace: argocd
    spec:
      description: ShopLite workloads
      sourceRepos:
        - https://github.com/<you>/shoplite-gitops.git
        - registry-1.docker.io/bitnamicharts
      destinations:
        - server: https://kubernetes.default.svc
          namespace: shoplite-*
      clusterResourceWhitelist:
        - group: ""
          kind: Namespace
  5. 5

    One Application per environment

    Move the Applications into argocd/apps/. web-dev now points at apps/web/overlays/dev with project: shoplite. web-prod is the same with the prod overlay. Prod keeps selfHeal but you may leave prune off there at first, which is a common, cautious choice.

    shoplite-gitops/argocd/apps/web-prod.yamlwhole fileyaml
    apiVersion: argoproj.io/v1alpha1
    kind: Application
    metadata:
      name: web-prod
      namespace: argocd
    spec:
      project: shoplite
      source:
        repoURL: https://github.com/<you>/shoplite-gitops.git
        targetRevision: main
        path: apps/web/overlays/prod
      destination:
        server: https://kubernetes.default.svc
        namespace: shoplite-prod
      syncPolicy:
        automated: { prune: false, selfHeal: true }
        syncOptions: [CreateNamespace=true]
  6. 6

    A third-party Helm chart, values in Git

    ShopLite's cart cache uses Redis. Argo renders the chart itself (helm template), so there's no Tiller and no helm install state: the Helm release doesn't show in helm list, and Argo tracks the resources instead. Pin targetRevision to an exact chart version. The Stateful Systems course covers running Redis properly.

    shoplite-gitops/argocd/apps/redis-dev.yamlwhole fileyaml
    apiVersion: argoproj.io/v1alpha1
    kind: Application
    metadata:
      name: redis-dev
      namespace: argocd
    spec:
      project: shoplite
      source:
        repoURL: registry-1.docker.io/bitnamicharts
        chart: redis
        targetRevision: 22.0.7
        helm:
          valuesObject:
            architecture: standalone
            auth: { enabled: true, existingSecret: redis-auth }
            master:
              persistence: { size: 1Gi }
      destination:
        server: https://kubernetes.default.svc
        namespace: shoplite-dev
      syncPolicy:
        automated: { prune: true, selfHeal: true }
  7. 7

    The root Application (app of apps)

    The root watches argocd/ recursively: projects and apps. From now on, the only thing you ever kubectl apply is this file, once per cluster. Delete the old Applications you applied by hand; the root recreates them from Git.

    shoplite-gitops/argocd/root.yamlwhole fileyaml
    apiVersion: argoproj.io/v1alpha1
    kind: Application
    metadata:
      name: root
      namespace: argocd
    spec:
      project: default
      source:
        repoURL: https://github.com/<you>/shoplite-gitops.git
        targetRevision: main
        path: argocd
        directory:
          recurse: true
          exclude: root.yaml
      destination:
        server: https://kubernetes.default.svc
        namespace: argocd
      syncPolicy:
        automated: { prune: true, selfHeal: true }
    terminal
    $ git add . && git commit -m 'kustomize overlays, project, app-of-apps' && git push
    kubectl apply -f argocd/root.yaml
    argocd app list
    ── expected output ──
    NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH
    root https://kubernetes.default.svc argocd default Synced Healthy
    redis-dev https://kubernetes.default.svc shoplite-dev shoplite Synced Healthy
    web-dev https://kubernetes.default.svc shoplite-dev shoplite Synced Healthy
    web-prod https://kubernetes.default.svc shoplite-prod shoplite Synced Healthy
  8. 8

    CI commits the tag instead of deploying

    The last job of the app repo's release workflow (CI/CD course, GitHub Actions). It uses a deploy key or GitHub App token with write access to ONLY the config repo, so there are no cluster credentials anywhere in CI. The commit message links back to the source commit for traceability.

    shoplite-web/.github/workflows/release.ymladd to fileyaml
      bump-dev:
        needs: build-and-push          # image shoplite-web:sha-${{ github.sha }} already pushed + signed
        runs-on: ubuntu-latest
        steps:
          - uses: actions/checkout@v5
            with:
              repository: <you>/shoplite-gitops
              token: ${{ secrets.GITOPS_REPO_TOKEN }}
          - uses: imranismail/setup-kustomize@v2
          - name: Set dev image tag
            run: |
              cd apps/web/overlays/dev
              kustomize edit set image nginxdemos/hello=ghcr.io/<you>/shoplite-web:sha-${{ github.sha }}
              git config user.name "shoplite-ci"
              git config user.email "ci@shoplite.dev"
              git commit -am "web(dev): deploy ${{ github.sha }} from ${{ github.repository }}"
              git push
  9. 9

    Promote to prod with a PR

    Promotion copies the tested tag from dev to prod. A branch-protection rule on main requires an approval for changes under apps/*/overlays/prod/ (CODEOWNERS), so deploys to prod get the same review as code (Git course, branch protection).

    terminal
    $ git switch -c promote-web-0.4
    cd apps/web/overlays/prod && kustomize edit set image nginxdemos/hello=nginxdemos/hello:0.4 && cd -
    git commit -am 'web(prod): promote 0.4' && git push -u origin promote-web-0.4
    gh pr create --fill --reviewer shoplite/platform
    ── expected output ──
    https://github.com/<you>/shoplite-gitops/pull/12

Checkpoint — you should now have

  • ✓argocd app list shows root, web-dev, web-prod, and redis-dev, all Synced and Healthy.
  • ✓Dev runs 0.4 and prod runs 0.3 until the promotion PR merges.
  • ✓CI has no kubeconfig; it can only write to the config repo.
  • ✓Deleting web-dev in the UI brings it back within seconds, because the root recreates it.

Part 4

Break it on purpose

Make each change, run the command, and read the error before revealing the diagnosis. Recognising these messages on sight is what makes you fast on a real team. Undo the change afterwards.

Break #1

Deploy outside the AppProject's fence

Change web-prod's destination namespace to kube-system and push.

terminal
$ argocd app get web-prod
── what you'll see ──
CONDITION MESSAGE
InvalidSpecError application destination server 'https://kubernetes.default.svc' and namespace 'kube-system' do not match any of the allowed destinations in project 'shoplite'

Break #2

CI loop: the config repo triggers the app build

Put the Kubernetes manifests in the APP repo and have CI commit the new tag to that same repo on every build.

terminal
$ gh run list --limit 4
── what you'll see ──
STATUS TITLE WORKFLOW EVENT
✓ web(dev): deploy 8c1e2f4 release push
✓ web(dev): deploy 3f2a1c9 release push
✓ web(dev): deploy 91be7d0 release push
✓ feat: add wishlist release push

Part 5

Interview questions from this mission

01

How do you handle multiple environments in GitOps?

02

What is the app-of-apps pattern?

03

Kustomize or Helm with Argo CD?

Before you stop

Clean up

terminal
$ # keep the cluster for Stage 2, or:
kind delete cluster --name shoplite
0/4 · 0%