Command Palette

Search for a command to run...

Hectal

Mission 2.2 · Stage 2 — GitOps at Scale: Secrets, Rollouts, and Many Clusters

Every Deployment Strategy, with Argo Rollouts

Goal: ShopLite web ships by canary with automated Prometheus analysis and auto-rollback, and the checkout service ships blue-green with a preview stage.

70 min Free 7 steps 2 break-it drills

By the end of this mission

  • Compare recreate, rolling, blue-green, canary, A/B, shadow, and feature-flag releases
  • Convert a Deployment to an Argo Rollout
  • Run a canary with traffic steps and pauses
  • Gate promotion on metrics with an AnalysisTemplate, and roll back automatically
  • Run a blue-green release with a preview Service

Part 1

Understand it first

The strategies, from simplest to safest

RECREATE: stop all old pods, start new ones. Downtime, but never two versions at once (useful when versions can't coexist). ROLLING UPDATE (Kubernetes default): replace pods a few at a time. No downtime, but no traffic control: after the rollout, 100% of users are on the new version, and a bad release hits everyone before you notice.

BLUE-GREEN: run the full new version (green) next to the old (blue), test green privately, then switch all traffic at once; rollback is switching back. Instant, but needs double capacity during the switch. CANARY: send a small share of traffic (5%, then 25%, then 50%) to the new version, watch error rate and latency, and continue or abort. Limits the blast radius of a bad release to a fraction of users for a few minutes.

A/B TESTING routes by user attributes (header, cookie, region) rather than percentage, and it's a product experiment more than a safety tool. SHADOW (mirroring) copies real traffic to the new version and discards its responses, testing under real load with zero user impact (writes need care). FEATURE FLAGS decouple deploy from release: code ships dark and is enabled per user in the app (LaunchDarkly, OpenFeature, Unleash). Mature teams combine canary deploys with flags.

Argo Rollouts

A Rollout is a drop-in replacement for a Deployment (same pod template) with a strategy.canary or strategy.blueGreen block. Without a traffic router, canary weights are approximated by pod counts (1 of 10 pods ≈ 10%). With a traffic router (NGINX Ingress, Gateway API, Istio, ALB), weights are exact and independent of replica counts.

ANALYSIS turns 'watch the dashboards' into code: an AnalysisTemplate runs Prometheus (or Datadog, CloudWatch, a Job) queries during the rollout, and if the success condition fails, the Rollout aborts and shifts traffic back automatically. This is the SLO thinking from the SRE course applied at deploy time.

Canary with analysisdiagram
Rendering diagram…

Part 2

Your project after this mission · 7 files change

shoplite-gitops/
  • shoplite-gitops/
    • apps/
      • checkout/
        • base/
          • rollout.yamlnew
      • web/
        • base/
          • analysis-success-rate.yamlnew
          • deployment.yamldeleted
          • ingress.yamlnew
          • rollout.yamlnew
          • service.yamlmodified
    • argocd/
      • apps/
        • argo-rollouts.yamlnew

Part 3

Build it, step by step

  1. 1

    Install the controller and kubectl plugin

    Add an Application for the argo-rollouts chart (https://argoproj.github.io/argo-helm) into namespace argo-rollouts, like ESO in Mission 2.1. The kubectl plugin gives you a live terminal view of rollouts. Argo CD understands Rollout health natively.

    terminal
    $ brew install argoproj/tap/kubectl-argo-rollouts # or download from GitHub releases
    kubectl -n argo-rollouts get pods
    ── expected output ──
    NAME READY STATUS
    argo-rollouts-6d8f9c7b5-x4k2p 1/1 Running
  2. 2

    Two Services: stable and canary

    The Rollout rewrites these Services' selectors (adding a pod-template-hash) so each one points at exactly one version, and the Ingress controller splits traffic between them. Duplicate service.yaml into web-stable and web-canary with identical selectors.

    shoplite-gitops/apps/web/base/service.yamlwhole fileyaml
    apiVersion: v1
    kind: Service
    metadata:
      name: web-stable
    spec:
      selector: { app: web }
      ports: [{ port: 80, targetPort: 80 }]
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: web-canary
    spec:
      selector: { app: web }
      ports: [{ port: 80, targetPort: 80 }]
  3. 3

    The Ingress that receives weighted traffic

    You only write the stable Ingress. Rollouts creates a canary Ingress next to it with NGINX's canary-weight annotation and updates the weight at each step.

    shoplite-gitops/apps/web/base/ingress.yamlwhole fileyaml
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: web
    spec:
      ingressClassName: nginx
      rules:
        - host: shop.local
          http:
            paths:
              - path: /
                pathType: Prefix
                backend:
                  service: { name: web-stable, port: { number: 80 } }
  4. 4

    Define 'healthy' as a query

    Success rate of the canary's requests, from the NGINX Ingress metrics you scraped in the Observability course. Checked every minute; three failures abort. inconclusive results (no traffic yet) pause rather than fail.

    shoplite-gitops/apps/web/base/analysis-success-rate.yamlwhole fileyaml
    apiVersion: argoproj.io/v1alpha1
    kind: AnalysisTemplate
    metadata:
      name: success-rate
    spec:
      args:
        - name: canary-service
      metrics:
        - name: success-rate
          interval: 1m
          failureLimit: 3
          successCondition: len(result) == 0 || result[0] >= 0.99
          provider:
            prometheus:
              address: http://prometheus.monitoring:9090
              query: |
                sum(rate(nginx_ingress_controller_requests{service="{{args.canary-service}}",status!~"5.."}[2m]))
                /
                sum(rate(nginx_ingress_controller_requests{service="{{args.canary-service}}"}[2m]))
  5. 5

    Replace the Deployment with a Rollout

    The pod template is unchanged from the Deployment. Only kind and strategy differ. Background analysis runs from step 1; the steps shift traffic 10 → 30 → 60 → 100 with pauses. pause: {} with no duration waits for a human to promote, which is useful before the final step in prod.

    shoplite-gitops/apps/web/base/rollout.yamlwhole fileyaml
    apiVersion: argoproj.io/v1alpha1
    kind: Rollout
    metadata:
      name: web
    spec:
      replicas: 4
      revisionHistoryLimit: 3
      selector:
        matchLabels: { app: web }
      template:
        metadata:
          labels: { app: web }
        spec:
          containers:
            - name: web
              image: nginxdemos/hello:0.3
              ports: [{ containerPort: 80 }]
              readinessProbe: { httpGet: { path: /, port: 80 } }
              resources:
                requests: { cpu: 50m, memory: 32Mi }
                limits: { memory: 64Mi }
      strategy:
        canary:
          stableService: web-stable
          canaryService: web-canary
          trafficRouting:
            nginx:
              stableIngress: web
          analysis:
            templates: [{ templateName: success-rate }]
            startingStep: 1
            args:
              - name: canary-service
                value: shoplite-prod-web-canary-80
          steps:
            - setWeight: 10
            - pause: { duration: 2m }
            - setWeight: 30
            - pause: { duration: 3m }
            - setWeight: 60
            - pause: { duration: 5m }
  6. 6

    Ship a canary and watch it

    Bump the tag in the overlay and push, exactly as before, because the GitOps flow doesn't change. Rollouts takes over once Argo applies the new spec.

    terminal
    $ cd apps/web/overlays/prod && kustomize edit set image nginxdemos/hello=nginxdemos/hello:0.4 && cd -
    git commit -am 'web(prod): canary 0.4' && git push
    kubectl argo rollouts get rollout web -n shoplite-prod --watch
    ── expected output ──
    Name: web
    Status: ॥ Paused
    Message: CanaryPauseStep
    Strategy: Canary
    Step: 1/6
    SetWeight: 10
    ActualWeight: 10
    Images: nginxdemos/hello:0.3 (stable)
    nginxdemos/hello:0.4 (canary)
    Replicas:
    Desired: 4
    Current: 5
     
    ⟳ web Rollout ॥ Paused
    ├──# revision:2
    │ ├──⧉ web-6c7d8e9f5 ReplicaSet ✔ Healthy canary
    │ └──α web-6c7d8e9f5-2 AnalysisRun ◌ Running
    └──# revision:1
    └──⧉ web-5b6c7d8e4 ReplicaSet ✔ Healthy stable
  7. 7

    Blue-green for checkout

    Checkout talks to the payment provider, and the team wants the entire new version tested before any real user sees it. Blue-green gives a preview Service for smoke tests (prePromotionAnalysis), then an instant switch. The old ReplicaSet stays up for scaleDownDelaySeconds so rollback is instant too.

    shoplite-gitops/apps/checkout/base/rollout.yamladd to fileyaml
      strategy:
        blueGreen:
          activeService: checkout-active       # real users
          previewService: checkout-preview     # QA + smoke tests
          autoPromotionEnabled: false          # a human (or analysis) promotes
          prePromotionAnalysis:
            templates: [{ templateName: smoke-test }]
          scaleDownDelaySeconds: 600
    terminal
    $ kubectl argo rollouts promote checkout -n shoplite-prod
    ── expected output ──
    rollout 'checkout' promoted

Checkpoint — you should now have

  • ✓A tag bump in Git produces a stepped canary visible in kubectl argo rollouts get.
  • ✓The AnalysisRun shows success-rate measurements at each step.
  • ✓Checkout's new version is reachable on the preview Service before promotion.

Part 4

Break it on purpose

Make each change, run the command, and read the error before revealing the diagnosis. Recognising these messages on sight is what makes you fast on a real team. Undo the change afterwards.

Break #1

Ship a version that returns errors

Deploy an image that answers 500 on a share of requests (for example ghcr.io/<you>/shoplite-web:faulty), and generate traffic with hey -z 5m http://shop.local/.

terminal
$ kubectl argo rollouts get rollout web -n shoplite-prod
── what you'll see ──
Status: ✖ Degraded
Message: RolloutAborted: Rollout aborted update to revision 3: Metric "success-rate" assessed Failed due to failed (3) > failureLimit (3)
SetWeight: 0
ActualWeight: 0
Images: nginxdemos/hello:0.4 (stable)

Break #2

Analysis with no traffic

Run the canary in dev where nobody sends requests.

terminal
$ kubectl -n shoplite-dev get analysisrun
── what you'll see ──
NAME STATUS AGE
web-7d8e9f6a5-3-1 Inconclusive 4m

Part 5

Interview questions from this mission

01

Blue-green vs canary: when do you pick which?

02

What does automated canary analysis need to work well?

03

How are feature flags different from canary deployments?

0/4 · 0%