Mission 2.2 · Stage 2 — GitOps at Scale: Secrets, Rollouts, and Many Clusters
Every Deployment Strategy, with Argo Rollouts
Goal: ShopLite web ships by canary with automated Prometheus analysis and auto-rollback, and the checkout service ships blue-green with a preview stage.
By the end of this mission
- Compare recreate, rolling, blue-green, canary, A/B, shadow, and feature-flag releases
- Convert a Deployment to an Argo Rollout
- Run a canary with traffic steps and pauses
- Gate promotion on metrics with an AnalysisTemplate, and roll back automatically
- Run a blue-green release with a preview Service
Part 1
Understand it first
The strategies, from simplest to safest
RECREATE: stop all old pods, start new ones. Downtime, but never two versions at once (useful when versions can't coexist). ROLLING UPDATE (Kubernetes default): replace pods a few at a time. No downtime, but no traffic control: after the rollout, 100% of users are on the new version, and a bad release hits everyone before you notice.
BLUE-GREEN: run the full new version (green) next to the old (blue), test green privately, then switch all traffic at once; rollback is switching back. Instant, but needs double capacity during the switch. CANARY: send a small share of traffic (5%, then 25%, then 50%) to the new version, watch error rate and latency, and continue or abort. Limits the blast radius of a bad release to a fraction of users for a few minutes.
A/B TESTING routes by user attributes (header, cookie, region) rather than percentage, and it's a product experiment more than a safety tool. SHADOW (mirroring) copies real traffic to the new version and discards its responses, testing under real load with zero user impact (writes need care). FEATURE FLAGS decouple deploy from release: code ships dark and is enabled per user in the app (LaunchDarkly, OpenFeature, Unleash). Mature teams combine canary deploys with flags.
Argo Rollouts
A Rollout is a drop-in replacement for a Deployment (same pod template) with a strategy.canary or strategy.blueGreen block. Without a traffic router, canary weights are approximated by pod counts (1 of 10 pods ≈ 10%). With a traffic router (NGINX Ingress, Gateway API, Istio, ALB), weights are exact and independent of replica counts.
ANALYSIS turns 'watch the dashboards' into code: an AnalysisTemplate runs Prometheus (or Datadog, CloudWatch, a Job) queries during the rollout, and if the success condition fails, the Rollout aborts and shifts traffic back automatically. This is the SLO thinking from the SRE course applied at deploy time.
Part 2
Your project after this mission · 7 files change
- shoplite-gitops/
- apps/
- checkout/
- base/
- rollout.yamlnew
- web/
- base/
- analysis-success-rate.yamlnew
- deployment.yamldeleted
- ingress.yamlnew
- rollout.yamlnew
- service.yamlmodified
- argocd/
- apps/
- argo-rollouts.yamlnew
Part 3
Build it, step by step
- 1
Install the controller and kubectl plugin
Add an Application for the
argo-rolloutschart (https://argoproj.github.io/argo-helm) into namespaceargo-rollouts, like ESO in Mission 2.1. The kubectl plugin gives you a live terminal view of rollouts. Argo CD understands Rollout health natively.terminal$ brew install argoproj/tap/kubectl-argo-rollouts # or download from GitHub releaseskubectl -n argo-rollouts get pods── expected output ──NAME READY STATUSargo-rollouts-6d8f9c7b5-x4k2p 1/1 Running - 2
Two Services: stable and canary
The Rollout rewrites these Services' selectors (adding a pod-template-hash) so each one points at exactly one version, and the Ingress controller splits traffic between them. Duplicate
service.yamlintoweb-stableandweb-canarywith identical selectors.shoplite-gitops/apps/web/base/service.yamlwhole fileyaml apiVersion: v1 kind: Service metadata: name: web-stable spec: selector: { app: web } ports: [{ port: 80, targetPort: 80 }] --- apiVersion: v1 kind: Service metadata: name: web-canary spec: selector: { app: web } ports: [{ port: 80, targetPort: 80 }] - 3
The Ingress that receives weighted traffic
You only write the stable Ingress. Rollouts creates a canary Ingress next to it with NGINX's
canary-weightannotation and updates the weight at each step.shoplite-gitops/apps/web/base/ingress.yamlwhole fileyaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: web spec: ingressClassName: nginx rules: - host: shop.local http: paths: - path: / pathType: Prefix backend: service: { name: web-stable, port: { number: 80 } } - 4
Define 'healthy' as a query
Success rate of the canary's requests, from the NGINX Ingress metrics you scraped in the Observability course. Checked every minute; three failures abort.
inconclusiveresults (no traffic yet) pause rather than fail.shoplite-gitops/apps/web/base/analysis-success-rate.yamlwhole fileyaml apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: success-rate spec: args: - name: canary-service metrics: - name: success-rate interval: 1m failureLimit: 3 successCondition: len(result) == 0 || result[0] >= 0.99 provider: prometheus: address: http://prometheus.monitoring:9090 query: | sum(rate(nginx_ingress_controller_requests{service="{{args.canary-service}}",status!~"5.."}[2m])) / sum(rate(nginx_ingress_controller_requests{service="{{args.canary-service}}"}[2m])) - 5
Replace the Deployment with a Rollout
The pod template is unchanged from the Deployment. Only
kindandstrategydiffer. Background analysis runs from step 1; the steps shift traffic 10 → 30 → 60 → 100 with pauses.pause: {}with no duration waits for a human to promote, which is useful before the final step in prod.shoplite-gitops/apps/web/base/rollout.yamlwhole fileyaml apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: web spec: replicas: 4 revisionHistoryLimit: 3 selector: matchLabels: { app: web } template: metadata: labels: { app: web } spec: containers: - name: web image: nginxdemos/hello:0.3 ports: [{ containerPort: 80 }] readinessProbe: { httpGet: { path: /, port: 80 } } resources: requests: { cpu: 50m, memory: 32Mi } limits: { memory: 64Mi } strategy: canary: stableService: web-stable canaryService: web-canary trafficRouting: nginx: stableIngress: web analysis: templates: [{ templateName: success-rate }] startingStep: 1 args: - name: canary-service value: shoplite-prod-web-canary-80 steps: - setWeight: 10 - pause: { duration: 2m } - setWeight: 30 - pause: { duration: 3m } - setWeight: 60 - pause: { duration: 5m } - 6
Ship a canary and watch it
Bump the tag in the overlay and push, exactly as before, because the GitOps flow doesn't change. Rollouts takes over once Argo applies the new spec.
terminal$ cd apps/web/overlays/prod && kustomize edit set image nginxdemos/hello=nginxdemos/hello:0.4 && cd -git commit -am 'web(prod): canary 0.4' && git pushkubectl argo rollouts get rollout web -n shoplite-prod --watch── expected output ──Name: webStatus: ॥ PausedMessage: CanaryPauseStepStrategy: CanaryStep: 1/6SetWeight: 10ActualWeight: 10Images: nginxdemos/hello:0.3 (stable)nginxdemos/hello:0.4 (canary)Replicas:Desired: 4Current: 5⟳ web Rollout ॥ Paused├──# revision:2│ ├──⧉ web-6c7d8e9f5 ReplicaSet ✔ Healthy canary│ └──α web-6c7d8e9f5-2 AnalysisRun ◌ Running└──# revision:1└──⧉ web-5b6c7d8e4 ReplicaSet ✔ Healthy stable - 7
Blue-green for checkout
Checkout talks to the payment provider, and the team wants the entire new version tested before any real user sees it. Blue-green gives a preview Service for smoke tests (
prePromotionAnalysis), then an instant switch. The old ReplicaSet stays up forscaleDownDelaySecondsso rollback is instant too.shoplite-gitops/apps/checkout/base/rollout.yamladd to fileyaml strategy: blueGreen: activeService: checkout-active # real users previewService: checkout-preview # QA + smoke tests autoPromotionEnabled: false # a human (or analysis) promotes prePromotionAnalysis: templates: [{ templateName: smoke-test }] scaleDownDelaySeconds: 600terminal$ kubectl argo rollouts promote checkout -n shoplite-prod── expected output ──rollout 'checkout' promoted
Checkpoint — you should now have
- ✓A tag bump in Git produces a stepped canary visible in
kubectl argo rollouts get. - ✓The AnalysisRun shows success-rate measurements at each step.
- ✓Checkout's new version is reachable on the preview Service before promotion.
Part 4
Break it on purpose
Make each change, run the command, and read the error before revealing the diagnosis. Recognising these messages on sight is what makes you fast on a real team. Undo the change afterwards.
Break #1
Ship a version that returns errors
Deploy an image that answers 500 on a share of requests (for example ghcr.io/<you>/shoplite-web:faulty), and generate traffic with hey -z 5m http://shop.local/.
Break #2
Analysis with no traffic
Run the canary in dev where nobody sends requests.
Part 5
Interview questions from this mission
Blue-green vs canary: when do you pick which?
What does automated canary analysis need to work well?
How are feature flags different from canary deployments?