Guide G4 · DevOps path
Infrastructure Testing and Policy as Code
The testing pyramid for infrastructure: formatting and static analysis, Terraform native tests with mocks, Terratest integration tests, Helm and Kubernetes manifest tests, OPA/Rego and Conftest policies, and testing the pipelines themselves.
Start here
The mental model
Application developers wouldn't merge code without tests, yet infrastructure code that can delete a database often ships with nothing but a human glancing at a plan. Infrastructure testing applies the same pyramid: many fast, cheap checks at the bottom (formatting, linting, policy on the plan), fewer slower ones in the middle (unit tests with mocked providers), and a few expensive end-to-end tests at the top (really create it, check it works, destroy it).
POLICY AS CODE is the 'rules' layer of that pyramid: 'no public S3 buckets', 'no pods running as root', 'every resource has an owner tag', written as code, tested, and enforced automatically in CI and in the cluster instead of in review meetings.
Go deeper
How it works inside
01The pyramid
STATIC (seconds, every commit): terraform fmt -check, terraform validate, TFLint (provider-aware rules like invalid instance types), security/misconfiguration scanners (Checkov, Trivy config), helm lint, kubeconform (schema-validate Kubernetes YAML), actionlint and yamllint for pipelines. PLAN-TIME POLICY: Conftest/OPA or Checkov against terraform show -json output, and kyverno test for Kubernetes policies. UNIT: terraform test with command = plan and mocked providers; helm-unittest for chart templates. INTEGRATION/E2E: terraform test with command = apply, or Terratest in Go, which creates real resources in a sandbox account, asserts on them, and destroys them; helm test and chart-testing (ct install) in a kind cluster.
02OPA and Rego
OPEN POLICY AGENT is a general-purpose policy engine; policies are written in REGO, a declarative query language. Given an input document (a Terraform plan as JSON, a Kubernetes object, an API request), rules produce decisions such as a set of deny messages. CONFTEST runs Rego policies against files in CI. GATEKEEPER runs them as Kubernetes admission control (the Rego-based alternative to Kyverno, Platform course Mission 0.4). The same policy language across Terraform, Kubernetes, and APIs is OPA's main advantage.
03Testing pipelines and charts
Pipelines are code too: lint them (actionlint, gitlab-ci-lint, Jenkins' declarative linter), run GitHub Actions locally with act for fast iteration, pin third-party actions by SHA, and test reusable workflows from a sample repo (CI/CD course). For Helm, helm template | kubeconform catches invalid output, helm-unittest asserts rendered values for given inputs, and chart-testing installs changed charts into a throwaway kind cluster and runs helm test hooks.
Do it
Hands-on lab
- 1
Static checks in one go
Run these locally and in CI on every PR (pre-commit hooks make them automatic).
terminal$ terraform fmt -check -recursive && terraform -chdir=envs/dev validatetflint --recursivecheckov -d . --quiet --compact | tail -3helm template charts/checkout | kubeconform -strict -summary── expected output ──Success! The configuration is valid.Check: CKV_AWS_144: "Ensure that S3 bucket has cross-region replication enabled"FAILED for resource: aws_s3_bucket.uploadsSummary: 14 resources found in 1 file - Valid: 14, Invalid: 0, Errors: 0, Skipped: 0 - 2
A Terraform unit test with a mocked provider
terraform test(Terraform 1.7+ for mocks) runs.tftest.hclfiles. Withmock_provider, no AWS credentials or resources are needed, so it's fast and safe on every PR. It asserts that the module computes what you expect.modules/network/tests/network.tftest.hclwhole filehcl mock_provider "aws" {} variables { name = "shoplite-test" cidr = "10.50.0.0/16" az_count = 3 } run "creates_one_private_subnet_per_az" { command = plan assert { condition = length(aws_subnet.private) == 3 error_message = "expected one private subnet per AZ" } } run "one_nat_gateway_per_az" { command = plan assert { condition = length(aws_nat_gateway.this) == var.az_count error_message = "NAT gateways must be per-AZ (no single point of failure)" } }terminal$ terraform -chdir=modules/network init -backend=false && terraform -chdir=modules/network test── expected output ──tests/network.tftest.hcl... in progressrun "creates_one_private_subnet_per_az"... passrun "one_nat_gateway_per_az"... passtests/network.tftest.hcl... tearing downtests/network.tftest.hcl... passSuccess! 2 passed, 0 failed. - 3
Policy as code with Rego and Conftest
Deny public S3 bucket ACLs and require an
ownertag on every resource that supports tags, evaluated against the JSON plan. It fails the pipeline before anything is applied.policy/terraform.regowhole filerego package main import rego.v1 resources := [r | some r in input.resource_changes; r.change.after != null] deny contains msg if { some r in resources r.type == "aws_s3_bucket_acl" r.change.after.acl in {"public-read", "public-read-write"} msg := sprintf("%s: public S3 ACLs are not allowed", [r.address]) } deny contains msg if { some r in resources tags := object.get(r.change.after, "tags_all", null) tags != null not tags.owner msg := sprintf("%s: missing required tag 'owner'", [r.address]) }terminal$ terraform plan -out tf.plan && terraform show -json tf.plan > plan.jsonconftest test plan.json -p policy/── expected output ──FAIL - plan.json - main - aws_s3_bucket.uploads: missing required tag 'owner'FAIL - plan.json - main - aws_s3_bucket_acl.uploads: public S3 ACLs are not allowed2 tests, 0 passed, 0 warnings, 2 failures, 0 exceptions - 4
An integration test with Terratest
Terratest (Go) applies real infrastructure in a sandbox account, checks it from the outside, and always destroys it (
defer). Run it nightly or on module releases, not on every commit, because it's slow and costs money.test/network_test.gowhole filego package test import ( "testing" "github.com/gruntwork-io/terratest/modules/aws" "github.com/gruntwork-io/terratest/modules/terraform" "github.com/stretchr/testify/assert" ) func TestNetworkModule(t *testing.T) { opts := terraform.WithDefaultRetryableErrors(t, &terraform.Options{ TerraformDir: "../modules/network", Vars: map[string]interface{}{"name": "tt-network", "cidr": "10.60.0.0/16", "az_count": 2}, }) defer terraform.Destroy(t, opts) terraform.InitAndApply(t, opts) vpcID := terraform.Output(t, opts, "vpc_id") subnets := aws.GetSubnetsForVpc(t, vpcID, "ap-south-1") assert.Len(t, subnets, 4) // 2 public + 2 private }terminal$ cd test && go test -v -timeout 30m -run TestNetworkModule── expected output ──--- PASS: TestNetworkModule (312.44s)PASSok github.com/shoplite/infra/test 312.461s
Operate it
Knobs that matter
| Setting | Default | What it does | When to change it |
|---|---|---|---|
| Where each test runs | — | Static + policy + unit on every PR; integration nightly/on release. | Keep PR feedback under ~5 minutes or people stop waiting for it. |
| Policy severity | fail | Whether a rule blocks or warns. | Introduce new rules as warnings (warn in Conftest), then promote to deny. |
| Sandbox account for Terratest | — | Where real test resources are created. | Separate account with budgets and a nightly cleanup (aws-nuke) for anything tests leak. |
3am practice
Failure drills
Each drill is a real failure mode. Read the scenario and the output, decide what's wrong, then reveal the diagnosis.
Drill #1
Tests pass, production breaks
Every PR check is green, but applying a change to prod fails: an RDS parameter group family doesn't exist for the engine version.
The bigger picture
Connects to
Prove it
Interview questions
How do you test infrastructure as code?
What is policy as code, and OPA vs Kyverno?