Rollback Strategies in CI/CD

In one line

Every rollback mechanism this course has covered — kubectl rollout undo, helm rollback, redeploying a previous artifact — is a genuine option; a pipeline's real job is deciding, automatically or with a human, exactly when to use one.

0/4 · 0%

Key ideas

01
This topic ties together every rollback mechanism this collection of courses has already covered, from the CI/CD PIPELINE's own point of view — Kubernetes' kubectl rollout undo (that course's Phase 6.3), Helm's helm rollback (that course's Phase 7.3), and Phase 5's own artifact-repository-based 'redeploy the previous known-good version' are all genuine rollback OPTIONS; this topic is about a pipeline actually deciding WHEN to use one.
02
An AUTOMATIC ROLLBACK TRIGGER is a pipeline (or a deployment tool's own built-in mechanism) that detects a genuine problem POST-DEPLOYMENT and initiates a rollback with NO human decision required — Topic 6.2's canary analysis stage failing is exactly this: the pipeline itself decides, based on real observed metrics, that a rollback is warranted, and executes it automatically.
03
A MANUAL rollback, by contrast, requires a human to notice a problem (often via monitoring/alerting, or a user report) and explicitly TRIGGER the rollback themselves — genuinely slower than an automatic trigger, but appropriate when the 'is this actually a problem requiring rollback' judgment call is genuinely too nuanced for a purely metrics-based automatic check to make reliably.
04
A genuinely important operational practice: the ROLLBACK PROCESS ITSELF should be tested and rehearsed BEFORE it's ever needed for real — a rollback mechanism that's never actually been exercised (Kubernetes' course, Phase 6.3, and Git's course, Phase 5.3's reflog rehearsal both made this exact point) is a genuine unknown risk precisely at the moment you can least afford a surprise: during an active incident.
05
MTTR (Mean Time To Recovery/Restore, a genuine metric the original DevOps roadmap's own SRE section names directly) measures how quickly a team can actually recover from a failure — a fast, well-rehearsed, ideally AUTOMATIC rollback mechanism is one of the single most direct, effective ways to improve this metric, since it removes both human detection delay and human execution delay from the recovery path entirely.
06
The genuinely mature target most real teams should aim for: automatic rollback for problems a pipeline can reliably DETECT itself (Topic 6.2's canary metrics check), combined with a fast, well-rehearsed MANUAL rollback path (a single, well-tested, one-command action) for the genuinely broader class of problems that require human judgment to first recognize as a real problem at all.

Code & diagrams

RollbackDecisionTreediagram

Automatic where the pipeline can reliably judge it; manual, but fast and rehearsed, everywhere else.

Rendering diagram…

rollback-pipeline-job.ymlmarkdown

A single, well-tested, one-command manual rollback — exactly the kind worth rehearsing before it's ever needed for real.

rollback:
  stage: rollback
  when: manual                # a human decides this is needed
  script:
    - echo "Rolling back to the last known-good version: $LAST_GOOD_VERSION"
    # the SAME artifact-retrieval principle as Phase 5.1 — redeploy, don't rebuild
    - ./deploy.sh production my-registry/myapp:$LAST_GOOD_VERSION
  environment:
    name: production
    action: prepare              # tracked distinctly from a normal forward deployment

# This job should be tested regularly — e.g. rehearsed in staging on a schedule —
# so the FIRST time it's ever run for real isn't during an actual incident

Explain it without notes

Why is a canary analysis stage failing (Topic 6.2) a genuinely good candidate for an automatic rollback trigger, while 'a customer emailed support saying something feels off' typically is not?

Why does MTTR specifically improve so directly from having a fast, automatic (or well-rehearsed manual) rollback mechanism?

Practice

For a hypothetical production system, identify one specific metric-based condition that would be a genuinely good candidate for an automatic rollback trigger, and one type of problem that would genuinely still need human judgment first.

If you have access to a real deployment pipeline, actually rehearse its rollback process once, deliberately, in a non-production environment — and note anything about the process that was slower or more confusing than expected.

Trade-offs

↔
Automatic rollback triggers are genuinely fast and remove human delay entirely, but an overly aggressive or poorly-tuned automatic trigger risks rolling back a deployment based on a genuinely temporary, harmless blip (a brief metric spike that would have resolved itself) — tuning exactly how sensitive an automatic trigger should be is a genuine, ongoing balance between avoiding real damage from a bad deployment and avoiding unnecessary, disruptive rollbacks triggered by noise rather than a genuine problem.

Done when you can

I can distinguish when an automatic rollback trigger is appropriate versus when human judgment is genuinely still needed.
I understand why rehearsing a rollback process before it's needed for real is a genuinely important practice.
I can explain how a fast, reliable rollback mechanism directly improves MTTR.