Command Palette

Search for a command to run...

Hectal

Mission 4.1 · Stage 4 — Data & Secrets

RDS Postgres in the Database Subnets

Goal: A private, encrypted Postgres 16 instance in the database subnets, reachable only from the app security group, with its master password generated and stored by AWS — not by you.

45 min (RDS takes ~8 minutes to create) ~$0.02/hour for db.t4g.micro single-AZ + storage 6 steps 2 break-it drills

By the end of this mission

  • Create a DB subnet group, parameter group, and RDS instance
  • Set production-vs-dev behaviour (Multi-AZ, deletion protection, final snapshots) from variables
  • Let RDS manage the master password in Secrets Manager
  • Connect to a private database through Session Manager port forwarding

Part 1

Understand it first

What Terraform owns vs what the database owns

Terraform manages the database INSTANCE: size, storage, network placement, backups, encryption, parameters. It does not manage what's INSIDE the database (tables, rows, application users). Schemas are an application concern, handled by migrations that ship with the app. Mixing them (running SQL from Terraform) couples schema changes to infrastructure applies and is rarely worth it.

Never let Terraform know the password

The classic mistake is password = var.db_password: the password is then in tfvars or CI variables, in every plan's inputs, and in state in plain text forever. manage_master_user_password = true asks RDS to generate the password, store it in Secrets Manager, and rotate it on a schedule. Terraform only ever sees the secret's ARN. Mission 4.2 wires that ARN into ECS.

Dev and prod differ in the dangerous settings

Settings like deletion_protection, skip_final_snapshot, multi_az, and apply_immediately are where environments should differ. Dev: single-AZ, no deletion protection, no final snapshot, apply changes immediately, so it's cheap and disposable. Prod: Multi-AZ, deletion protection on, a final snapshot on delete, and changes held for the maintenance window. Deriving them from var.environment means prod can't accidentally get dev's settings.

Pinning only the major version

With auto_minor_version_upgrade (the default), AWS upgrades 16.4 to 16.6 during maintenance windows. If your code says engine_version = "16.4", the next plan tries to DOWNGRADE, which fails. Specifying just "16" means 'any 16.x'. The provider compares against the major version, and engine_version_actual tells you the running minor.

Where the database sitsdiagram
Rendering diagram…

Part 2

Your project after this mission · 3 files change

shoplite/
  • infra/
    • alb.tf
    • autoscaling.tf
    • backend.tf
    • database.tfnew
    • ecr.tf
    • ecs.tf
    • iam.tf
    • locals.tf
    • logs.tf
    • network.tf
    • outputs.tfmodified
    • probe.tf
    • providers.tf
    • refactors.tf
    • security.tf
    • storage.tf
    • terraform.tfvars
    • variables.tfmodified
    • versions.tf

Part 3

Build it, step by step

  1. 1

    Database variables

    Only size and HA are variables; the safety settings are derived from the environment in the resource itself.

    infra/variables.tfadd to filehcl
    variable "db_instance_class" {
      type    = string
      default = "db.t4g.micro"
    }
    
    variable "db_multi_az" {
      description = "Synchronous standby in a second AZ. Required in prod."
      type        = bool
      default     = false
    }
  2. 2

    Subnet group and parameter group

    The subnet group tells RDS which subnets it may use, here the database tier with no internet route. The parameter group forces TLS for every connection (rds.force_ssl) and logs queries slower than 500 ms. Parameter groups can't be renamed or have their family changed in place, hence name_prefix with create_before_destroy, the same pattern as security groups.

    infra/database.tfwhole filehcl
    resource "aws_db_subnet_group" "main" {
      name       = "${local.name_prefix}-db"
      subnet_ids = [for s in aws_subnet.database : s.id]
    }
    
    resource "aws_db_parameter_group" "postgres16" {
      name_prefix = "${local.name_prefix}-pg16-"
      family      = "postgres16"
    
      parameter {
        name  = "rds.force_ssl"
        value = "1"
      }
    
      parameter {
        name  = "log_min_duration_statement"
        value = "500"
      }
    
      lifecycle {
        create_before_destroy = true
      }
    }
  3. 3

    The instance

    Read it top to bottom: encrypted gp3 storage that autoscales up to 100 GB; manage_master_user_password instead of a password; placed in the database subnets with the DB security group from Mission 2.3; not publicly accessible; seven days of point-in-time backups; and the environment-dependent safety settings.

    infra/database.tfadd to filehcl
    locals {
      is_prod = var.environment == "prod"
    }
    
    resource "aws_db_instance" "main" {
      identifier     = "${local.name_prefix}-db"
      engine         = "postgres"
      engine_version = "16"
      instance_class = var.db_instance_class
    
      allocated_storage     = 20
      max_allocated_storage = 100
      storage_type          = "gp3"
      storage_encrypted     = true
    
      db_name                     = "shoplite"
      username                    = "shoplite_admin"
      manage_master_user_password = true
    
      db_subnet_group_name   = aws_db_subnet_group.main.name
      vpc_security_group_ids = [aws_security_group.db.id]
      parameter_group_name   = aws_db_parameter_group.postgres16.name
      publicly_accessible    = false
      multi_az               = var.db_multi_az
    
      backup_retention_period = 7
      copy_tags_to_snapshot   = true
    
      deletion_protection       = local.is_prod
      skip_final_snapshot       = !local.is_prod
      final_snapshot_identifier = "${local.name_prefix}-db-final"
      apply_immediately         = !local.is_prod
    }
  4. 4

    Outputs — an ARN, not a password

    master_user_secret is a list with one element holding the secret's ARN. That's all Terraform knows.

    infra/outputs.tfadd to filehcl
    output "db_endpoint" {
      value = aws_db_instance.main.address
    }
    
    output "db_master_secret_arn" {
      value = aws_db_instance.main.master_user_secret[0].secret_arn
    }
  5. 5

    Apply — and be patient

    RDS creation takes 5–10 minutes. Terraform polls and prints progress. Everything else in the plan finishes in seconds.

    terminal
    $ terraform apply
    ── expected output ──
    Plan: 3 to add, 0 to change, 0 to destroy.
    ...
    aws_db_instance.main: Still creating... [5m0s elapsed]
    aws_db_instance.main: Still creating... [7m30s elapsed]
    aws_db_instance.main: Creation complete after 7m52s [id=db-ABCDEFGHIJKLMNOPQRSTUVWXYZ]
     
    Apply complete! Resources: 3 added, 0 changed, 0 destroyed.
     
    Outputs:
    ...
    db_endpoint = "shoplite-dev-db.c1x2y3z4a5b6.ap-south-1.rds.amazonaws.com"
    db_master_secret_arn = "arn:aws:secretsmanager:ap-south-1:123456789012:secret:rds!db-7f3e...-AbCdEf"
  6. 6

    Connect through a Session Manager tunnel

    The database has no public endpoint and only accepts the app security group. The probe instance from Mission 2.4 uses that group, so turn it on and forward a local port through it to RDS. No SSH, no bastion, no open ports. Fetch the password with YOUR credentials (not Terraform's state) and connect with psql from your laptop.

    terminal
    $ terraform apply -var enable_probe=true -auto-approve >/dev/null
    aws ssm start-session --target $(terraform output -raw probe_instance_id) \
    --document-name AWS-StartPortForwardingSessionToRemoteHost \
    --parameters "host=$(terraform output -raw db_endpoint),portNumber=5432,localPortNumber=15432" &
    export PGPASSWORD=$(aws secretsmanager get-secret-value --secret-id "$(terraform output -raw db_master_secret_arn)" --query SecretString --output text | jq -r .password)
    psql "host=localhost port=15432 dbname=shoplite user=shoplite_admin sslmode=require" -c 'select version();'
    ── expected output ──
    Starting session with SessionId: priya-0f1e2d...
    Port 15432 opened for sessionId priya-0f1e2d....
    version
    -------------------------------------------------------------------------------
    PostgreSQL 16.6 on aarch64-unknown-linux-gnu, compiled by gcc (GCC) 12.4.0, 64-bit
    (1 row)
    Turn the probe off again afterwards: `terraform apply -var enable_probe=false`.

Checkpoint — you should now have

  • ✓An encrypted, non-public Postgres 16 instance runs in the database subnets.
  • ✓Terraform code and state contain no password; terraform output db_master_secret_arn shows only an ARN.
  • ✓You connected with psql through an SSM tunnel using the secret from Secrets Manager.
  • ✓In dev: deletion protection off and final snapshot skipped. In prod these flip automatically.

Part 4

Break it on purpose

Make each change, run the command, and read the error before revealing the diagnosis. Recognising these messages on sight is what makes you fast on a real team. Undo the change afterwards.

Break #1

Rename the database identifier

Change identifier to "${local.name_prefix}-postgres" and run terraform plan. Do NOT apply.

terminal
$ terraform plan
── what you'll see ──
# aws_db_instance.main must be replaced
-/+ resource "aws_db_instance" "main" {
~ address = "shoplite-dev-db.c1x2y3z4a5b6.ap-south-1.rds.amazonaws.com" -> (known after apply)
~ identifier = "shoplite-dev-db" -> "shoplite-dev-postgres" # forces replacement
...
}
 
Plan: 1 to add, 0 to change, 1 to destroy.

Break #2

Upgrade the major version by just changing the number

Set engine_version = "17" and apply.

terminal
$ terraform apply
── what you'll see ──
# aws_db_instance.main will be updated in-place
~ resource "aws_db_instance" "main" {
~ engine_version = "16" -> "17"
...
╷
│ Error: updating RDS DB Instance (shoplite-dev-db): operation error RDS:
│ ModifyDBInstance, https response error StatusCode: 400, api error
│ InvalidParameterCombination: The AllowMajorVersionUpgrade flag must be
│ present when upgrading to a new major version.
╵

Part 5

Interview questions from this mission

01

How do you create an RDS database with Terraform without the master password ending up in state?

02

Which RDS settings should differ between dev and prod, and how do you keep them from being mixed up?

03

Why set engine_version = "16" rather than "16.4"?

Before you stop

Clean up

terminal
$ terraform apply -var enable_probe=false
── expected output ──
Apply complete! Resources: 0 added, 0 changed, 4 destroyed.
0/4 · 0%