Command Palette

Search for a command to run...

Hectal

Mission 7.1 · Stage 7 — Team Workflow

Plan on Pull Request, Apply on Merge — with OIDC

Goal: Every pull request touching infra gets a plan for dev and prod posted as a comment; merging to main applies dev automatically and prod after an approval — and no AWS key exists anywhere.

60 min Free (GitHub Actions minutes on public repos / free tier) 5 steps 2 break-it drills

By the end of this mission

  • Let GitHub Actions assume AWS roles through OIDC with no stored secrets
  • Separate a read-only plan role from a write apply role
  • Post plans on pull requests and gate prod applies with GitHub Environments
  • Prevent concurrent pipeline runs from fighting over state

Part 1

Understand it first

Why CI owns applies

When applies happen from CI, every change to real infrastructure has a pull request (the why), a reviewed plan (the what), an approver (the who), and a log (the when). Laptops stop needing write access to prod at all. That removes the single largest source of 'someone ran apply with uncommitted code' incidents.

OIDC: short-lived credentials, no secrets

GitHub can issue each workflow run a signed identity token that says, for example, 'this is repo you/shoplite, running for a pull request' or 'running in the prod environment'. AWS IAM trusts GitHub's OIDC provider, and a role's trust policy matches on that token's sub claim. The workflow exchanges the token for temporary credentials that last about an hour. No access key is ever stored in GitHub. This is the pattern from the AWS course, Topic 1.4, applied here.

Because the trust policy can match the context, you can give pull requests (untrusted code) only a READ-ONLY role that can plan, and allow the WRITE role only for runs in the protected prod environment on main.

Plan on PR, re-plan on merge

The PR plan is for REVIEW. By the time the PR merges, other PRs may have merged and state may have changed, so the saved PR plan could be stale. On merge, the pipeline plans again against the latest state and applies that exact saved plan in the same job. Serialising runs per environment with a concurrency group ensures two merges never apply at once; the state lock (Mission 1.2) is the last line of defence, not the first.

The pipelinediagram
Rendering diagram…

Part 2

Your project after this mission · 2 files change

shoplite/
  • .github/
    • workflows/
      • terraform.ymlnew
  • bootstrap/
    • github-oidc.tfnew
    • main.tf
  • infra/
    • envs/
      • dev.s3.tfbackend
      • dev.tfvars
      • prod.s3.tfbackend
      • prod.tfvars
    • database.tf
    • ecs.tf
    • main.tf
    • providers.tf
    • security.tf
    • tf
  • modules/
    • ecs-service/
      • main.tf
      • variables.tf
    • network/
      • main.tf

Part 3

Build it, step by step

  1. 1

    Trust GitHub from AWS (in the bootstrap config, per account)

    One OIDC provider per account, and two roles. The PLAN role trusts any pull request from your repo and gets read-only access plus state access (it needs to write the lock file). The DEPLOYER role, which you've been assuming yourself since Mission 6.3, now also trusts GitHub, but ONLY for runs in the dev or prod GitHub environment. Apply this with the bootstrap config in each account.

    bootstrap/github-oidc.tfwhole filehcl
    variable "github_repo" {
      type    = string
      default = "you/shoplite"
    }
    
    variable "environment" {
      type = string # dev or prod — which account this bootstrap runs in
    }
    
    resource "aws_iam_openid_connect_provider" "github" {
      url            = "https://token.actions.githubusercontent.com"
      client_id_list = ["sts.amazonaws.com"]
    }
    
    data "aws_iam_policy_document" "ci_plan_trust" {
      statement {
        actions = ["sts:AssumeRoleWithWebIdentity"]
        principals {
          type        = "Federated"
          identifiers = [aws_iam_openid_connect_provider.github.arn]
        }
        condition {
          test     = "StringEquals"
          variable = "token.actions.githubusercontent.com:aud"
          values   = ["sts.amazonaws.com"]
        }
        condition {
          test     = "StringEquals"
          variable = "token.actions.githubusercontent.com:sub"
          values   = ["repo:${var.github_repo}:pull_request"]
        }
      }
    }
    
    resource "aws_iam_role" "ci_plan" {
      name               = "shoplite-ci-plan"
      assume_role_policy = data.aws_iam_policy_document.ci_plan_trust.json
    }
    
    resource "aws_iam_role_policy_attachment" "ci_plan_readonly" {
      role       = aws_iam_role.ci_plan.name
      policy_arn = "arn:aws:iam::aws:policy/ReadOnlyAccess"
    }
    
    data "aws_iam_policy_document" "ci_plan_state" {
      statement {
        sid       = "StateReadAndLock"
        actions   = ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket"]
        resources = [aws_s3_bucket.tfstate.arn, "${aws_s3_bucket.tfstate.arn}/*"]
      }
      statement {
        sid       = "NeverReadSecretValues"
        effect    = "Deny"
        actions   = ["secretsmanager:GetSecretValue"]
        resources = ["*"]
      }
    }
    
    resource "aws_iam_role_policy" "ci_plan_state" {
      role   = aws_iam_role.ci_plan.id
      policy = data.aws_iam_policy_document.ci_plan_state.json
    }
    
    # The deployer role (created during account setup) gets an extra trust statement:
    #   sub = "repo:${var.github_repo}:environment:${var.environment}"
    # so only jobs running in the matching GitHub Environment can assume it.
  2. 2

    Create GitHub Environments

    In the repo settings, create environments dev and prod. Give prod REQUIRED REVIEWERS (you, or your team) and restrict it to the main branch. Store each environment's account ID as an environment variable, AWS_ACCOUNT_ID, and also as repository variables DEV_ACCOUNT_ID and PROD_ACCOUNT_ID for the PR plan matrix. These aren't secrets; OIDC needs none.

  3. 3

    The workflow

    Read it in three parts. plan runs for pull requests as a matrix over both environments with the read-only role, and posts each plan as a PR comment. apply-dev runs on push to main, re-plans, and applies that saved plan. apply-prod needs apply-dev and runs in the prod environment, so it waits for approval. concurrency serialises runs per environment and never cancels a running apply.

    .github/workflows/terraform.ymlwhole fileyaml
    name: terraform
    
    on:
      pull_request:
        paths: ["infra/**", "modules/**"]
      push:
        branches: [main]
        paths: ["infra/**", "modules/**"]
    
    permissions:
      id-token: write      # OIDC token for AWS
      contents: read
      pull-requests: write # plan comments
    
    env:
      TF_IN_AUTOMATION: "true"
      TF_INPUT: "false"
      AWS_REGION: ap-south-1
    
    jobs:
      plan:
        if: github.event_name == 'pull_request'
        runs-on: ubuntu-latest
        strategy:
          fail-fast: false
          matrix:
            env: [dev, prod]
        concurrency: plan-${{ matrix.env }}-${{ github.head_ref }}
        steps:
          - uses: actions/checkout@v4
          - uses: hashicorp/setup-terraform@v3
            with:
              terraform_version: 1.13.3
              terraform_wrapper: false
          - uses: aws-actions/configure-aws-credentials@v4
            with:
              role-to-assume: arn:aws:iam::${{ vars[format('{0}_ACCOUNT_ID', matrix.env)] }}:role/shoplite-ci-plan
              aws-region: ${{ env.AWS_REGION }}
          - name: fmt
            run: terraform fmt -check -recursive
          - name: plan
            working-directory: infra
            run: |
              ./tf ${{ matrix.env }} validate
              ./tf ${{ matrix.env }} plan -lock-timeout=5m -no-color -out=tfplan
              terraform show -no-color tfplan > plan.txt
          - name: comment
            uses: actions/github-script@v7
            with:
              script: |
                const plan = require('fs').readFileSync('infra/plan.txt', 'utf8');
                const summary = plan.split('\n').find(l => l.startsWith('Plan:') || l.startsWith('No changes')) ?? '';
                const body = `### \`${{ matrix.env }}\` — ${summary}\n<details><summary>Full plan</summary>\n\n\`\`\`\n${plan.slice(-60000)}\n\`\`\`\n</details>`;
                await github.rest.issues.createComment({ ...context.repo, issue_number: context.issue.number, body });
    
      apply-dev:
        if: github.event_name == 'push'
        runs-on: ubuntu-latest
        environment: dev
        concurrency:
          group: apply-dev
          cancel-in-progress: false
        steps: &apply-steps
          - uses: actions/checkout@v4
          - uses: hashicorp/setup-terraform@v3
            with:
              terraform_version: 1.13.3
              terraform_wrapper: false
          - uses: aws-actions/configure-aws-credentials@v4
            with:
              role-to-assume: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/shoplite-deployer
              aws-region: ${{ env.AWS_REGION }}
          - name: plan and apply
            working-directory: infra
            run: |
              ./tf ${{ github.job == 'apply-prod' && 'prod' || 'dev' }} plan -lock-timeout=5m -out=tfplan
              terraform apply -lock-timeout=5m tfplan
    
      apply-prod:
        if: github.event_name == 'push'
        needs: apply-dev
        runs-on: ubuntu-latest
        environment: prod     # required reviewers → the job waits for approval
        concurrency:
          group: apply-prod
          cancel-in-progress: false
        steps: *apply-steps
  4. 4

    Open a pull request and watch

    Change something small: bump api_memory to 1024 in envs/dev.tfvars. The PR gets two comments. Dev shows one task definition replacement and a service update; prod shows No changes. Reviewers approve based on those comments, not on reading HCL diffs alone.

    terminal
    $ git switch -c api-memory && sed -i 's/^api_memory.*//' infra/envs/dev.tfvars && echo 'api_memory = 1024' >> infra/envs/dev.tfvars
    git commit -am 'dev: api memory 1024' && git push -u origin api-memory && gh pr create --fill
    ── expected output ──
    https://github.com/you/shoplite/pull/42
     
    # PR comments posted by the workflow:
    ### `dev` — Plan: 1 to add, 1 to change, 1 to destroy.
    ### `prod` — No changes. Your infrastructure matches the configuration.
  5. 5

    Merge: dev applies, prod waits for you

    After merging, apply-dev runs immediately. apply-prod shows 'Waiting for review' until a required reviewer approves in the Actions UI. Here prod has nothing to do, but the gate is always there. From now on, stop applying from your laptop.

    terminal
    $ gh pr merge 42 --squash && gh run watch
    ── expected output ──
    ✓ apply-dev Apply complete! Resources: 1 added, 1 changed, 1 destroyed.
    * apply-prod Waiting for review: prod needs approval to start deploying changes.

Checkpoint — you should now have

  • ✓GitHub has no AWS access keys; both jobs authenticate through OIDC.
  • ✓PRs get a plan comment per environment, produced with a read-only role.
  • ✓Merges apply dev automatically; prod waits for a required reviewer.
  • ✓Applies re-plan on main and apply that saved plan; runs are serialised per environment.

Part 4

Break it on purpose

Make each change, run the command, and read the error before revealing the diagnosis. Recognising these messages on sight is what makes you fast on a real team. Undo the change afterwards.

Break #1

Make the PR role able to apply

Temporarily point the plan job at the shoplite-deployer role instead of shoplite-ci-plan and push to the PR.

terminal
$ (GitHub Actions log — configure-aws-credentials step)
── what you'll see ──
Error: Could not assume role with OIDC: Not authorized to perform
sts:AssumeRoleWithWebIdentity

Break #2

Let a second run cancel an apply

Set cancel-in-progress: true on apply-dev, then merge two PRs a few seconds apart.

terminal
$ (GitHub Actions — first run)
── what you'll see ──
aws_ecs_service.this: Modifying...
Error: The operation was canceled.
 
(next run)
╷
│ Error: Error acquiring the state lock
│ ...
│ Who: runner@fv-az812-431
│ Operation: OperationTypeApply
╵

Part 5

Interview questions from this mission

01

How would you run Terraform in CI/CD on AWS securely?

02

Why re-plan on merge instead of applying the plan created for the pull request?

03

Why should PR jobs never have write credentials?

0/4 · 0%