Command Palette

Search for a command to run...

Hectal

Mission 4.3 · Stage 4 — Data & Secrets

Guardrails: prevent_destroy, Preconditions, and check Blocks

Goal: Terraform refuses any plan that would destroy the ShopLite database, refuses to deploy prod without Multi-AZ, and warns after every apply if the live API isn't healthy.

35 min Free 4 steps 3 break-it drills

By the end of this mission

  • Use all four lifecycle arguments and know their limits
  • Validate assumptions at plan time with preconditions and postconditions
  • Add continuous health assertions with check blocks
  • Recreate a broken resource on purpose with -replace

Part 1

Understand it first

The lifecycle meta-arguments

create_before_destroy (you've used it for security groups, target groups, and parameter groups) changes replacement order. ignore_changes (Mission 3.4) hands attributes to other controllers. prevent_destroy = true makes any plan that would destroy the resource an ERROR, whether from terraform destroy, a replacement, or deleted code. replace_triggered_by forces replacement when another resource or attribute changes.

Important limit: lifecycle arguments are processed before variables are evaluated, so they must be LITERALS. prevent_destroy = var.environment == "prod" is not allowed. For per-environment protection, combine a literal prevent_destroy with AWS-side protections like deletion_protection that can be variables.

Preconditions and postconditions

Variable validation checks a single input in isolation. PRECONDITIONS (in a resource's lifecycle) check assumptions across several values before the resource is planned, for example 'if environment is prod, multi_az must be true'. POSTCONDITIONS check what came back, for example 'the AMI this data source found really is arm64'. A failure stops the plan with your error message, pointing at the exact condition.

They turn tribal knowledge ('never deploy prod single-AZ') into rules the tool enforces on every run, for every person.

check blocks: assertions that warn

A check block (Terraform 1.5+) runs assertions at the end of every plan and apply, optionally with its own scoped data source. Unlike preconditions, a failing check produces a WARNING and doesn't block anything. That's right for things outside Terraform's control that you still want to know about, such as 'the public API returns 200'. After an apply that technically succeeded, a failed check tells you the service isn't actually healthy.

When each guardrail firesdiagram
Rendering diagram…

Part 2

Your project after this mission · 4 files change

shoplite/
  • infra/
    • alb.tf
    • autoscaling.tf
    • backend.tf
    • checks.tfnew
    • database.tfmodified
    • ecr.tf
    • ecs.tf
    • iam.tf
    • locals.tf
    • logs.tf
    • network.tf
    • outputs.tf
    • probe.tfmodified
    • providers.tf
    • refactors.tf
    • security.tf
    • storage.tf
    • terraform.tfvars
    • variables.tf
    • versions.tfmodified

Part 3

Build it, step by step

  1. 1

    Protect the database from Terraform itself

    Add prevent_destroy to the DB instance's lifecycle. It has to be a literal, so this applies to dev too. That's deliberate friction: to tear down dev you'll consciously remove the line first (see Clean up), and that removal shows up in review.

    infra/database.tfadd to filehcl

    Add inside resource "aws_db_instance" "main".

      lifecycle {
        prevent_destroy = true
    
        precondition {
          condition     = var.environment != "prod" || var.db_multi_az
          error_message = "Production databases must be Multi-AZ (set db_multi_az = true)."
        }
    
        precondition {
          condition     = var.environment != "prod" || !startswith(var.db_instance_class, "db.t")
          error_message = "Burstable (db.t*) instance classes aren't allowed in prod."
        }
      }
  2. 2

    Verify an assumption about a data source

    The probe's AMI lookup filters by name. If Amazon ever changed naming and the filter matched an x86 image, an ARM t4g instance would fail to launch with a confusing error. A postcondition turns that into a clear message at plan time.

    infra/probe.tfadd to filehcl

    Add inside data "aws_ami" "al2023_arm".

      lifecycle {
        postcondition {
          condition     = self.architecture == "arm64"
          error_message = "AMI lookup returned ${self.id} (${self.architecture}); expected an arm64 image for t4g instances."
        }
      }
  3. 3

    Add a live health check

    The http provider can make requests from within a check block. Add http = { source = "hashicorp/http", version = "~> 3.4" } to required_providers and run terraform init. The check calls the database health endpoint through the public URL, which covers the ALB, ECS, the app, secrets, and RDS in one request.

    infra/checks.tfwhole filehcl
    check "api_health" {
      data "http" "db_health" {
        url = "http://${aws_lb.main.dns_name}/healthz/db"
    
        retry {
          attempts = 3
        }
      }
    
      assert {
        condition     = data.http.db_health.status_code == 200
        error_message = "ShopLite /healthz/db returned ${data.http.db_health.status_code}: ${data.http.db_health.response_body}"
      }
    }
    terminal
    $ terraform init -upgrade >/dev/null && terraform apply
    ── expected output ──
    ...
    Plan: 0 to add, 0 to change, 0 to destroy.
    ...
    Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
    A passing check prints nothing. You'll see it fail in Break it.
  4. 4

    Rebuild something on purpose with -replace

    Occasionally a resource is broken in a way Terraform can't see, such as a corrupted instance or a stuck task definition. -replace=ADDRESS plans a replacement of exactly that resource. It's the modern substitute for the old terraform taint. Try it on the probe; it's safe.

    terminal
    $ terraform apply -var enable_probe=true -auto-approve >/dev/null
    terraform plan -var enable_probe=true -replace='aws_instance.probe[0]'
    ── expected output ──
    # aws_instance.probe[0] will be replaced, as requested
    -/+ resource "aws_instance" "probe" {
    ...
    Plan: 1 to add, 0 to change, 1 to destroy.

Checkpoint — you should now have

  • ✓The DB instance has prevent_destroy = true and two prod-only preconditions.
  • ✓The AMI data source has an arm64 postcondition.
  • ✓check "api_health" runs on every plan/apply (silent when healthy).
  • ✓You know -replace replaces the deprecated terraform taint.

Part 4

Break it on purpose

Make each change, run the command, and read the error before revealing the diagnosis. Recognising these messages on sight is what makes you fast on a real team. Undo the change afterwards.

Break #1

Try to replace the protected database

Repeat the rename from Mission 4.1 (identifier = "${local.name_prefix}-postgres") and run terraform plan.

terminal
$ terraform plan
── what you'll see ──
╷
│ Error: Instance cannot be destroyed
│
│ on database.tf line 28:
│ 28: resource "aws_db_instance" "main" {
│
│ Resource aws_db_instance.main has lifecycle.prevent_destroy set, but the
│ plan calls for this resource to be destroyed. To avoid this error and
│ continue with the plan, either disable lifecycle.prevent_destroy or reduce
│ the scope of the plan using the -target option.
╵

Break #2

Make prevent_destroy depend on the environment

Change it to prevent_destroy = var.environment == "prod" and run terraform validate.

terminal
$ terraform validate
── what you'll see ──
╷
│ Error: Variables not allowed
│
│ on database.tf line 54, in resource "aws_db_instance" "main":
│ 54: prevent_destroy = var.environment == "prod"
│
│ Variables may not be used here.
╵

Break #3

Watch a check fail

Break the app's database access in a way the deploy won't notice: remove the DB_HOST entry from the container's environment list and apply.

terminal
$ terraform apply
── what you'll see ──
...
Apply complete! Resources: 1 added, 1 changed, 1 destroyed.
╷
│ Warning: Check block assertion failed
│
│ on checks.tf line 10, in check "api_health":
│ 10: condition = data.http.db_health.status_code == 200
│ ├────────────────
│ │ data.http.db_health.status_code is 503
│
│ ShopLite /healthz/db returned 503: {"db":"not configured"}
╵

Part 5

Interview questions from this mission

01

What are the lifecycle meta-arguments and a use case for each?

02

What's the difference between variable validation, preconditions, and check blocks?

03

How do you force Terraform to recreate a resource that looks fine in state?

Before you stop

Clean up

terminal
$ # Tearing dev down now requires a deliberate edit:
# 1) remove prevent_destroy from aws_db_instance.main, 2) then:
terraform destroy
── expected output ──
Destroy complete!
Or leave it running; db.t4g.micro is ~$0.50/day. Put prevent_destroy back before the next apply.
0/4 · 0%