Command Palette

Search for a command to run...

Hectal

Guide G4 · DevOps path

Infrastructure Testing and Policy as Code

The testing pyramid for infrastructure: formatting and static analysis, Terraform native tests with mocks, Terratest integration tests, Helm and Kubernetes manifest tests, OPA/Rego and Conftest policies, and testing the pipelines themselves.

Advanced 55 min

Start here

The mental model

Application developers wouldn't merge code without tests, yet infrastructure code that can delete a database often ships with nothing but a human glancing at a plan. Infrastructure testing applies the same pyramid: many fast, cheap checks at the bottom (formatting, linting, policy on the plan), fewer slower ones in the middle (unit tests with mocked providers), and a few expensive end-to-end tests at the top (really create it, check it works, destroy it).

POLICY AS CODE is the 'rules' layer of that pyramid: 'no public S3 buckets', 'no pods running as root', 'every resource has an owner tag', written as code, tested, and enforced automatically in CI and in the cluster instead of in review meetings.

Go deeper

How it works inside

01The pyramid

STATIC (seconds, every commit): terraform fmt -check, terraform validate, TFLint (provider-aware rules like invalid instance types), security/misconfiguration scanners (Checkov, Trivy config), helm lint, kubeconform (schema-validate Kubernetes YAML), actionlint and yamllint for pipelines. PLAN-TIME POLICY: Conftest/OPA or Checkov against terraform show -json output, and kyverno test for Kubernetes policies. UNIT: terraform test with command = plan and mocked providers; helm-unittest for chart templates. INTEGRATION/E2E: terraform test with command = apply, or Terratest in Go, which creates real resources in a sandbox account, asserts on them, and destroys them; helm test and chart-testing (ct install) in a kind cluster.

The pyramiddiagram
Rendering diagram…

02OPA and Rego

OPEN POLICY AGENT is a general-purpose policy engine; policies are written in REGO, a declarative query language. Given an input document (a Terraform plan as JSON, a Kubernetes object, an API request), rules produce decisions such as a set of deny messages. CONFTEST runs Rego policies against files in CI. GATEKEEPER runs them as Kubernetes admission control (the Rego-based alternative to Kyverno, Platform course Mission 0.4). The same policy language across Terraform, Kubernetes, and APIs is OPA's main advantage.

03Testing pipelines and charts

Pipelines are code too: lint them (actionlint, gitlab-ci-lint, Jenkins' declarative linter), run GitHub Actions locally with act for fast iteration, pin third-party actions by SHA, and test reusable workflows from a sample repo (CI/CD course). For Helm, helm template | kubeconform catches invalid output, helm-unittest asserts rendered values for given inputs, and chart-testing installs changed charts into a throwaway kind cluster and runs helm test hooks.

Do it

Hands-on lab

  1. 1

    Static checks in one go

    Run these locally and in CI on every PR (pre-commit hooks make them automatic).

    terminal
    $ terraform fmt -check -recursive && terraform -chdir=envs/dev validate
    tflint --recursive
    checkov -d . --quiet --compact | tail -3
    helm template charts/checkout | kubeconform -strict -summary
    ── expected output ──
    Success! The configuration is valid.
    Check: CKV_AWS_144: "Ensure that S3 bucket has cross-region replication enabled"
    FAILED for resource: aws_s3_bucket.uploads
    Summary: 14 resources found in 1 file - Valid: 14, Invalid: 0, Errors: 0, Skipped: 0
  2. 2

    A Terraform unit test with a mocked provider

    terraform test (Terraform 1.7+ for mocks) runs .tftest.hcl files. With mock_provider, no AWS credentials or resources are needed, so it's fast and safe on every PR. It asserts that the module computes what you expect.

    modules/network/tests/network.tftest.hclwhole filehcl
    mock_provider "aws" {}
    
    variables {
      name     = "shoplite-test"
      cidr     = "10.50.0.0/16"
      az_count = 3
    }
    
    run "creates_one_private_subnet_per_az" {
      command = plan
      assert {
        condition     = length(aws_subnet.private) == 3
        error_message = "expected one private subnet per AZ"
      }
    }
    
    run "one_nat_gateway_per_az" {
      command = plan
      assert {
        condition     = length(aws_nat_gateway.this) == var.az_count
        error_message = "NAT gateways must be per-AZ (no single point of failure)"
      }
    }
    terminal
    $ terraform -chdir=modules/network init -backend=false && terraform -chdir=modules/network test
    ── expected output ──
    tests/network.tftest.hcl... in progress
    run "creates_one_private_subnet_per_az"... pass
    run "one_nat_gateway_per_az"... pass
    tests/network.tftest.hcl... tearing down
    tests/network.tftest.hcl... pass
     
    Success! 2 passed, 0 failed.
  3. 3

    Policy as code with Rego and Conftest

    Deny public S3 bucket ACLs and require an owner tag on every resource that supports tags, evaluated against the JSON plan. It fails the pipeline before anything is applied.

    policy/terraform.regowhole filerego
    package main
    
    import rego.v1
    
    resources := [r | some r in input.resource_changes; r.change.after != null]
    
    deny contains msg if {
      some r in resources
      r.type == "aws_s3_bucket_acl"
      r.change.after.acl in {"public-read", "public-read-write"}
      msg := sprintf("%s: public S3 ACLs are not allowed", [r.address])
    }
    
    deny contains msg if {
      some r in resources
      tags := object.get(r.change.after, "tags_all", null)
      tags != null
      not tags.owner
      msg := sprintf("%s: missing required tag 'owner'", [r.address])
    }
    terminal
    $ terraform plan -out tf.plan && terraform show -json tf.plan > plan.json
    conftest test plan.json -p policy/
    ── expected output ──
    FAIL - plan.json - main - aws_s3_bucket.uploads: missing required tag 'owner'
    FAIL - plan.json - main - aws_s3_bucket_acl.uploads: public S3 ACLs are not allowed
     
    2 tests, 0 passed, 0 warnings, 2 failures, 0 exceptions
  4. 4

    An integration test with Terratest

    Terratest (Go) applies real infrastructure in a sandbox account, checks it from the outside, and always destroys it (defer). Run it nightly or on module releases, not on every commit, because it's slow and costs money.

    test/network_test.gowhole filego
    package test
    
    import (
    	"testing"
    
    	"github.com/gruntwork-io/terratest/modules/aws"
    	"github.com/gruntwork-io/terratest/modules/terraform"
    	"github.com/stretchr/testify/assert"
    )
    
    func TestNetworkModule(t *testing.T) {
    	opts := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
    		TerraformDir: "../modules/network",
    		Vars:         map[string]interface{}{"name": "tt-network", "cidr": "10.60.0.0/16", "az_count": 2},
    	})
    	defer terraform.Destroy(t, opts)
    	terraform.InitAndApply(t, opts)
    
    	vpcID := terraform.Output(t, opts, "vpc_id")
    	subnets := aws.GetSubnetsForVpc(t, vpcID, "ap-south-1")
    	assert.Len(t, subnets, 4) // 2 public + 2 private
    }
    terminal
    $ cd test && go test -v -timeout 30m -run TestNetworkModule
    ── expected output ──
    --- PASS: TestNetworkModule (312.44s)
    PASS
    ok github.com/shoplite/infra/test 312.461s

Operate it

Knobs that matter

SettingDefaultWhat it doesWhen to change it
Where each test runs—Static + policy + unit on every PR; integration nightly/on release.Keep PR feedback under ~5 minutes or people stop waiting for it.
Policy severityfailWhether a rule blocks or warns.Introduce new rules as warnings (warn in Conftest), then promote to deny.
Sandbox account for Terratest—Where real test resources are created.Separate account with budgets and a nightly cleanup (aws-nuke) for anything tests leak.

3am practice

Failure drills

Each drill is a real failure mode. Read the scenario and the output, decide what's wrong, then reveal the diagnosis.

Drill #1

Tests pass, production breaks

Every PR check is green, but applying a change to prod fails: an RDS parameter group family doesn't exist for the engine version.

terminal
$ terraform apply
── what you'll see ──
Error: creating RDS DB Parameter Group (shoplite-prod-pg17): InvalidParameterValue: ParameterGroupFamily postgres17x is not a valid parameter group family

The bigger picture

Connects to

Prove it

Interview questions

01

How do you test infrastructure as code?

02

What is policy as code, and OPA vs Kyverno?