Flux, Promotion Pipelines, and Running GitOps in Production

1. 1

   The same app, the Flux way

   For comparison, here's what ShopLite web in dev looks like in a Flux repo. flux bootstrap github installs Flux and commits its own manifests into the repo, so Flux manages itself from the start.

   apiVersion: source.toolkit.fluxcd.io/v1
   kind: GitRepository
   metadata:
     name: shoplite-gitops
     namespace: flux-system
   spec:
     interval: 1m
     url: https://github.com/<you>/shoplite-gitops
     ref: { branch: main }
   ---
   apiVersion: kustomize.toolkit.fluxcd.io/v1
   kind: Kustomization
   metadata:
     name: web-dev
     namespace: flux-system
   spec:
     interval: 10m
     sourceRef: { kind: GitRepository, name: shoplite-gitops }
     path: ./apps/web/overlays/dev
     prune: true
     wait: true
     timeout: 3m
     dependsOn:
       - name: external-secrets

   terminalcopy cmd

   $ flux bootstrap github --owner=<you> --repository=shoplite-flux --path=clusters/dev

   flux get kustomizations

   ── expected output ──

   NAME REVISION SUSPENDED READY MESSAGE

   flux-system main@sha1:4e5f6a7b False True Applied revision: main@sha1:4e5f6a7b

   web-dev main@sha1:4e5f6a7b False True Applied revision: main@sha1:4e5f6a7b

2. 2

   SSO groups → Argo CD roles

   Policies are Casbin CSV lines: p, <role>, <resource>, <action>, <project>/<object>, allow. Developers can see and sync ShopLite apps but can't delete them or touch clusters; the local admin account gets disabled in argocd-cm (admin.enabled: "false") once SSO works.

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: argocd-rbac-cm
     namespace: argocd
   data:
     policy.default: role:readonly
     policy.csv: |
       p, role:shoplite-dev, applications, get,  shoplite/*, allow
       p, role:shoplite-dev, applications, sync, shoplite/*, allow
       p, role:shoplite-dev, logs,         get,  shoplite/*, allow
       g, shoplite-developers, role:shoplite-dev
       g, platform-team, role:admin
     scopes: "[groups]"

3. 3

   Tell the team when things go wrong

   Argo CD notifications ship with a trigger/template catalog. Subscribe an app (or all apps via defaults) to degraded and failed-sync triggers.

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: argocd-notifications-cm
     namespace: argocd
   data:
     service.slack: |
       token: $slack-token
     defaultTriggers: |
       - on-health-degraded
       - on-sync-failed
     template.app-degraded: |
       message: ":red_circle: {{.app.metadata.name}} is {{.app.status.health.status}} at {{.app.status.sync.revision | trunc 7}}"
     trigger.on-health-degraded: |
       - when: app.status.health.status == 'Degraded'
         send: [app-degraded]

4. 4

   Alert on GitOps itself

   A PrometheusRule (Observability course) for apps stuck unhealthy or out of sync. 'For 15 minutes' avoids paging on normal rollouts.

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: argocd
     namespace: monitoring
   spec:
     groups:
       - name: argocd
         rules:
           - alert: ArgoAppDegraded
             expr: argocd_app_info{health_status="Degraded"} == 1
             for: 15m
             labels: { severity: page }
             annotations:
               summary: "{{ $labels.name }} degraded for 15m"
           - alert: ArgoAppOutOfSync
             expr: argocd_app_info{sync_status="OutOfSync"} == 1
             for: 30m
             labels: { severity: ticket }

5. 5

   Rehearse a rebuild

   The DR drill: delete the dev cluster, recreate it, and time how long until everything is Healthy. Anything you had to do by hand that isn't in these four commands is a gap to close.

   terminalcopy cmd

   $ kind delete cluster --name shoplite && kind create cluster --name shoplite

   helm install argocd argo/argo-cd -n argocd --create-namespace --version 8.5.0

   kubectl apply -f argocd/root.yaml

   argocd app wait -l env=dev --health --timeout 900

   ── expected output ──

   ...

   Name: argocd/web-in-cluster

   Sync Status: Synced

   Health Status: Healthy

   (total time from empty cluster: 6m41s)

Make each change, run the command, and read the error before revealing the diagnosis. Recognising these messages on sight is what makes you fast on a real team. Undo the change afterwards.