Command Palette

Search for a command to run...

Hectal

Mission 2.2 · Stage 2 — Network

Internet Gateway, NAT, and Routing

Goal: Public subnets reach the internet through an internet gateway; private subnets reach out through NAT that you can switch between one gateway (dev), one per AZ (prod), or none (to stop paying); database subnets have no internet route at all.

45 min ~$0.06/hour per NAT gateway (~$1.35/day) — this mission shows how to switch it off 7 steps 2 break-it drills

By the end of this mission

  • Wire route tables and associations with for_each over existing resource maps
  • Build conditional infrastructure from boolean variables
  • Use depends_on for a real hidden dependency
  • Add an S3 gateway endpoint to keep S3 traffic off the NAT

Part 1

Understand it first

Conditional infrastructure

Dev and prod need different networks: dev wants one cheap NAT gateway, prod wants one per AZ so an AZ outage doesn't cut off the others (AWS course, Topic 3.2). You could write two configs, but that means two copies to keep in sync. Instead, express the difference as data: a list nat_azs that is empty, one AZ, or all AZs, depending on two booleans. Every NAT-related resource uses for_each over that list, so the same code produces zero, one, or N gateways.

The conditional expression cond ? a : b picks the value. Both branches must have compatible types, which is why the empty branch is [], a list like the others.

Iterating over another resource's map

for_each = aws_subnet.public is valid: a resource with for_each IS a map from key to object, and its keys (the AZ names) are known at plan time even though the subnet IDs aren't yet. So each association is keyed by AZ and uses each.value.id. That's the pattern to use instead of iterating over a list of IDs, which fails when the IDs are unknown (see Break it).

When depends_on is actually needed

A NAT gateway can only send traffic to the internet once the VPC has an attached internet gateway, but no argument of aws_nat_gateway references the internet gateway. Nothing in the references tells Terraform about that ordering. depends_on = [aws_internet_gateway.main] states the hidden dependency explicitly. It's the textbook case: use depends_on only when the dependency is real and invisible, never as a general 'just in case'.

Traffic paths after this missiondiagram
Rendering diagram…

Part 2

Your project after this mission · 3 files change

shoplite/
  • bootstrap/
    • backend.tf
    • main.tf
  • infra/
    • backend.tf
    • locals.tf
    • logs.tf
    • network.tfmodified
    • outputs.tf
    • providers.tf
    • refactors.tf
    • storage.tf
    • terraform.tfvarsmodified
    • variables.tfmodified
    • versions.tf
  • .gitignore

Part 3

Build it, step by step

  1. 1

    Add the NAT switches

    enable_nat_gateway = false removes NAT entirely, useful when you stop for the day, since NAT is the most expensive thing in this stage. single_nat_gateway picks between cheap dev and highly available prod.

    infra/variables.tfadd to filehcl
    variable "enable_nat_gateway" {
      description = "Give private subnets outbound internet access through NAT."
      type        = bool
      default     = true
    }
    
    variable "single_nat_gateway" {
      description = "true = one shared NAT (cheap, dev). false = one NAT per AZ (HA, prod)."
      type        = bool
      default     = true
    }
  2. 2

    Internet gateway and public routing

    One public route table shared by all public subnets. The association uses for_each = aws_subnet.public, keyed by AZ.

    infra/network.tfadd to filehcl
    resource "aws_internet_gateway" "main" {
      vpc_id = aws_vpc.main.id
      tags   = { Name = "${local.name_prefix}-igw" }
    }
    
    resource "aws_route_table" "public" {
      vpc_id = aws_vpc.main.id
      tags   = { Name = "${local.name_prefix}-public" }
    }
    
    resource "aws_route" "public_internet" {
      route_table_id         = aws_route_table.public.id
      destination_cidr_block = "0.0.0.0/0"
      gateway_id             = aws_internet_gateway.main.id
    }
    
    resource "aws_route_table_association" "public" {
      for_each = aws_subnet.public
    
      subnet_id      = each.value.id
      route_table_id = aws_route_table.public.id
    }
  3. 3

    Conditional NAT gateways and private routing

    local.nat_azs is the whole switch. Each private route table (one per AZ) routes to its own AZ's NAT when there's one per AZ, or to the single shared one otherwise. Routes live in separate aws_route resources (not inline in the route table), so they can be conditional.

    infra/network.tfadd to filehcl
    locals {
      nat_azs = !var.enable_nat_gateway ? [] : (
        var.single_nat_gateway ? [local.azs[0]] : local.azs
      )
    }
    
    resource "aws_eip" "nat" {
      for_each = toset(local.nat_azs)
    
      domain = "vpc"
      tags   = { Name = "${local.name_prefix}-nat-${each.key}" }
    }
    
    resource "aws_nat_gateway" "main" {
      for_each = toset(local.nat_azs)
    
      allocation_id = aws_eip.nat[each.key].id
      subnet_id     = aws_subnet.public[each.key].id
      tags          = { Name = "${local.name_prefix}-nat-${each.key}" }
    
      # NAT needs the IGW attached before it can route, but references nothing on it.
      depends_on = [aws_internet_gateway.main]
    }
    
    resource "aws_route_table" "private" {
      for_each = toset(local.azs)
    
      vpc_id = aws_vpc.main.id
      tags   = { Name = "${local.name_prefix}-private-${each.key}" }
    }
    
    resource "aws_route" "private_nat" {
      for_each = var.enable_nat_gateway ? toset(local.azs) : toset([])
    
      route_table_id         = aws_route_table.private[each.key].id
      destination_cidr_block = "0.0.0.0/0"
      nat_gateway_id         = aws_nat_gateway.main[var.single_nat_gateway ? local.azs[0] : each.key].id
    }
    
    resource "aws_route_table_association" "private" {
      for_each = aws_subnet.private
    
      subnet_id      = each.value.id
      route_table_id = aws_route_table.private[each.key].id
    }
  4. 4

    Keep S3 traffic off the NAT

    An S3 gateway endpoint is free and removes NAT data-processing charges for every image layer, upload, and log shipped to S3. The for expression collects all private route table IDs.

    Database subnets get no association at all, so they use the VPC's main route table, which only has the local route and therefore no path to the internet. That's intentional.

    infra/network.tfadd to filehcl
    resource "aws_vpc_endpoint" "s3" {
      vpc_id            = aws_vpc.main.id
      service_name      = "com.amazonaws.${var.aws_region}.s3"
      vpc_endpoint_type = "Gateway"
      route_table_ids   = [for rt in aws_route_table.private : rt.id]
    
      tags = { Name = "${local.name_prefix}-s3-endpoint" }
    }
  5. 5

    Plan and apply

    With the defaults (NAT on, single), the summary is: IGW, public route table plus route plus 2 associations, 1 EIP, 1 NAT, 2 private route tables, 2 private routes, 2 private associations, and the endpoint. The NAT gateway takes a minute or two to create.

    terminal
    $ terraform apply
    ── expected output ──
    ...
    Plan: 15 to add, 0 to change, 0 to destroy.
    ...
    aws_nat_gateway.main["ap-south-1a"]: Still creating... [1m20s elapsed]
    aws_nat_gateway.main["ap-south-1a"]: Creation complete after 1m34s [id=nat-0f1e2d3c4b5a69788]
    aws_route.private_nat["ap-south-1a"]: Creating...
    aws_route.private_nat["ap-south-1b"]: Creating...
    ...
    Apply complete! Resources: 15 added, 0 changed, 0 destroyed.
  6. 6

    Flip to prod-style HA, then read the plan (don't apply)

    Setting single_nat_gateway = false shows exactly what HA costs in resources: one more EIP and NAT, and the AZ-b route re-pointed to its own NAT. The plan is the cost review.

    terminal
    $ terraform plan -var single_nat_gateway=false
    ── expected output ──
    # aws_eip.nat["ap-south-1b"] will be created
    # aws_nat_gateway.main["ap-south-1b"] will be created
    # aws_route.private_nat["ap-south-1b"] will be updated in-place
    ~ resource "aws_route" "private_nat" {
    ~ nat_gateway_id = "nat-0f1e2d3c4b5a69788" -> (known after apply)
    ...
    }
     
    Plan: 2 to add, 1 to change, 0 to destroy.
  7. 7

    Know how to stop paying between sessions

    Put the switch in terraform.tfvars and turn it off whenever you stop working. Only the NAT, its EIP, and the private default routes are destroyed; the VPC, subnets, and everything else stay. Turn it back on (true) at the start of your next session.

    infra/terraform.tfvarswhole filehcl
    environment        = "dev"
    enable_nat_gateway = true # set to false when you stop for the day

Checkpoint — you should now have

  • ✓Public subnets route 0.0.0.0/0 to the internet gateway; private subnets route it to a NAT gateway.
  • ✓Database subnets have no internet route (they use the main route table).
  • ✓An S3 gateway endpoint is attached to both private route tables.
  • ✓You've read the plan for single_nat_gateway=false and know what HA adds.
  • ✓You know how to set enable_nat_gateway = false to stop NAT charges.

Part 4

Break it on purpose

Make each change, run the command, and read the error before revealing the diagnosis. Recognising these messages on sight is what makes you fast on a real team. Undo the change afterwards.

Break #1

for_each over values that don't exist yet

Rewrite the private association as for_each = toset([for s in aws_subnet.private : s.id]) with subnet_id = each.value, destroy the private associations and subnets (terraform destroy -target=aws_subnet.private), then run terraform plan.

terminal
$ terraform plan
── what you'll see ──
╷
│ Error: Invalid for_each argument
│
│ on network.tf line 88, in resource "aws_route_table_association" "private":
│ 88: for_each = toset([for s in aws_subnet.private : s.id])
│ ├────────────────
│ │ aws_subnet.private is object with 2 attributes
│
│ The "for_each" set includes values derived from resource attributes that
│ cannot be determined until apply, and so Terraform cannot determine the
│ full set of keys that will identify the instances of this resource.
│
│ When working with unknown values in for_each, it's better to use a map
│ value where the keys are defined statically in your configuration and
│ where only the values contain apply-time results.
╵

Break #2

Associate one subnet with two route tables

Add a second association that puts the ap-south-1a public subnet in the private route table: resource "aws_route_table_association" "oops" { subnet_id = aws_subnet.public["ap-south-1a"].id route_table_id = aws_route_table.private["ap-south-1a"].id }. Apply.

terminal
$ terraform apply
── what you'll see ──
aws_route_table_association.oops: Creating...
╷
│ Error: creating Route Table (rtb-0b7c...) Association: operation error EC2:
│ AssociateRouteTable, https response error StatusCode: 400, api error
│ Resource.AlreadyAssociated: the specified association for route table
│ rtb-0b7c... conflicts with an existing association
╵

Part 5

Interview questions from this mission

01

How would you let one Terraform configuration create a cheap dev network and a highly available prod network?

02

Give a legitimate use of depends_on.

03

Why do for_each keys have to be known at plan time?

Before you stop

Clean up

terminal
$ # Stopping for the day? Turn NAT off (keeps everything else):
terraform apply -var enable_nat_gateway=false
── expected output ──
Plan: 0 to add, 0 to change, 4 to destroy.
...
Apply complete! Resources: 0 added, 0 changed, 4 destroyed.
4 = the NAT gateway, its EIP, and the two private default routes. Set it back to true next session.
0/4 · 0%