Day 2: Drift Detection, Upgrades, Stuck Locks, and Teardown

1. 1

   Nightly drift detection

   Runs at 02:30 UTC per environment with the READ-ONLY plan role. It needs a trust statement for scheduled runs on main: add repo:you/shoplite:ref:refs/heads/main to the plan role's sub values. Exit code 2 opens an issue with the plan attached. Note the careful exit-code capture; see Break it for why.

   name: drift
   on:
     schedule: [{ cron: "30 2 * * *" }]
     workflow_dispatch: {}
   
   permissions:
     id-token: write
     contents: read
     issues: write
   
   jobs:
     drift:
       runs-on: ubuntu-latest
       strategy:
         fail-fast: false
         matrix: { env: [dev, prod] }
       steps:
         - uses: actions/checkout@v4
         - uses: hashicorp/setup-terraform@v3
           with: { terraform_version: 1.13.3, terraform_wrapper: false }
         - uses: aws-actions/configure-aws-credentials@v4
           with:
             role-to-assume: arn:aws:iam::${{ vars[format('{0}_ACCOUNT_ID', matrix.env)] }}:role/shoplite-ci-plan
             aws-region: ap-south-1
         - name: plan
           id: plan
           working-directory: infra
           run: |
             set +e
             ./tf ${{ matrix.env }} plan -detailed-exitcode -no-color -lock=false > plan.txt
             code=$?
             set -e
             echo "exitcode=$code" >> "$GITHUB_OUTPUT"
             [ "$code" -eq 1 ] && exit 1 || exit 0
         - name: open issue on drift
           if: steps.plan.outputs.exitcode == '2'
           env: { GH_TOKEN: "${{ github.token }}" }
           run: |
             gh issue create --title "Drift detected in ${{ matrix.env }} ($(date -u +%F))" \
               --label drift --body-file infra/plan.txt

2. 2

   Upgrade providers deliberately

   On a branch: re-select the newest allowed versions and record checksums for every platform the team and CI use. The lock-file diff shows exactly which versions changed. The PR's plans then show whether the upgrade changes anything real.

   terminalcopy cmd

   $ cd infra

   terraform init -upgrade -backend=false >/dev/null

   terraform providers lock -platform=linux_amd64 -platform=darwin_arm64 -platform=windows_amd64

   git diff --stat .terraform.lock.hcl

   ── expected output ──

   - Fetching hashicorp/aws 6.21.0 for linux_amd64...

   - Retrieved hashicorp/aws 6.21.0 for linux_amd64 (signed by HashiCorp)

   - Fetching hashicorp/aws 6.21.0 for darwin_arm64...

   ...

   Success! Terraform has updated the lock file.

    

   infra/.terraform.lock.hcl | 46 +++++++++++++++++++++++-----------------------

   1 file changed, 23 insertions(+), 23 deletions(-)

3. 3

   Tear it all down, in order

   When you're done with the course, remove everything so nothing keeps billing. First lift the guardrails: remove prevent_destroy from the database, and in prod set deletion_protection/final snapshots as you see fit. Then destroy consumers first, and bootstrap last, where you also remove prevent_destroy on the state bucket and empty its versions.

   terminalcopy cmd

   $ (cd analytics && terraform destroy -auto-approve)

   (cd infra && ./tf prod destroy) # if you applied prod

   (cd infra && ./tf dev destroy)

   # bootstrap last: remove prevent_destroy, empty all object versions, then:

   (cd bootstrap && terraform destroy)

   ── expected output ──

   Destroy complete! Resources: 1 destroyed.

   ...

   Destroy complete! Resources: 83 destroyed.

   ...

   Destroy complete! Resources: 11 destroyed.

Make each change, run the command, and read the error before revealing the diagnosis. Recognising these messages on sight is what makes you fast on a real team. Undo the change afterwards.