Autoscaling and Versioned Deploys

1. 1

   Stop Terraform from fighting the autoscaler

   Add a lifecycle block to the service. From now on, Terraform sets desired_count only when creating the service.

   Add inside resource "aws_ecs_service" "api".

     lifecycle {
       ignore_changes = [desired_count]
     }

2. 2

   Scaling bounds and policies

   resource_id has a fixed format for ECS: service/<cluster>/<service>. The request-count policy needs a resource_label built from the ALB and target group arn_suffix attributes, which exist for exactly this purpose.

   variable "api_min_count" {
     type    = number
     default = 2
   }
   
   variable "api_max_count" {
     type    = number
     default = 6
   }
   
   resource "aws_appautoscaling_target" "api" {
     service_namespace  = "ecs"
     scalable_dimension = "ecs:service:DesiredCount"
     resource_id        = "service/${aws_ecs_cluster.main.name}/${aws_ecs_service.api.name}"
     min_capacity       = var.api_min_count
     max_capacity       = var.api_max_count
   }
   
   resource "aws_appautoscaling_policy" "api_cpu" {
     name               = "${local.name_prefix}-api-cpu"
     policy_type        = "TargetTrackingScaling"
     service_namespace  = aws_appautoscaling_target.api.service_namespace
     scalable_dimension = aws_appautoscaling_target.api.scalable_dimension
     resource_id        = aws_appautoscaling_target.api.resource_id
   
     target_tracking_scaling_policy_configuration {
       target_value       = 60
       scale_out_cooldown = 30
       scale_in_cooldown  = 180
   
       predefined_metric_specification {
         predefined_metric_type = "ECSServiceAverageCPUUtilization"
       }
     }
   }
   
   resource "aws_appautoscaling_policy" "api_requests" {
     name               = "${local.name_prefix}-api-requests"
     policy_type        = "TargetTrackingScaling"
     service_namespace  = aws_appautoscaling_target.api.service_namespace
     scalable_dimension = aws_appautoscaling_target.api.scalable_dimension
     resource_id        = aws_appautoscaling_target.api.resource_id
   
     target_tracking_scaling_policy_configuration {
       target_value       = 500
       scale_out_cooldown = 30
       scale_in_cooldown  = 180
   
       predefined_metric_specification {
         predefined_metric_type = "ALBRequestCountPerTarget"
         resource_label         = "${aws_lb.main.arn_suffix}/${aws_lb_target_group.api.arn_suffix}"
       }
     }
   }

   terminalcopy cmd

   $ terraform apply

   ── expected output ──

   Plan: 3 to add, 0 to change, 0 to destroy.

   ...

   Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

3. 3

   Load it and watch it scale

   Use any HTTP load tool; hey is a single binary. Three minutes at 50 concurrent requests pushes requests per target well past 500. Scale-out takes a few minutes, because CloudWatch alarms need several datapoints. Stop the load and it scales back to 2 after the scale-in cooldown.

   terminalcopy cmd

   $ hey -z 4m -c 50 "$(terraform output -raw api_url)/products" > /dev/null &

   watch -n 30 "aws ecs describe-services --cluster shoplite-dev --services shoplite-dev-api --query 'services[0].[desiredCount,runningCount]' --output text"

   ── expected output ──

   2 2

   2 2

   4 2

   4 4

   6 6

4. 4

   Confirm Terraform leaves the count alone

   While the service is scaled out, plan. Without ignore_changes this would show desired_count = 6 -> 2; with it, nothing.

   terminalcopy cmd

   $ terraform plan

   ── expected output ──

   No changes. Your infrastructure matches the configuration.

5. 5

   Ship version 1.1.0

   A real change: the root endpoint reports which task served the request, so you can watch a rolling deploy happen. Bump package.json to 1.1.0, build, and push the new tag.

   Replace the final fallback line in the request handler.

   import os from "node:os";
   // ...
     return json(res, 200, { service: "shoplite-api", version, task: os.hostname() });

   terminalcopy cmd

   $ docker buildx build --platform linux/arm64 -t "$REPO:1.1.0" --push ../app

   ── expected output ──

   => pushing 123456789012.dkr.ecr.ap-south-1.amazonaws.com/shoplite/api:1.1.0

6. 6

   Deploy it: read the plan, then apply

   Exactly two changes: the task definition is replaced (new revision) and the service updated to use it. Nothing else. This is what every normal deploy plan should look like, and it's what Stage 7's pipeline expects. Run a curl loop in another terminal during the apply: responses switch from 1.0.0 to 1.1.0 without a single error.

   terminalcopy cmd

   $ terraform apply -var image_tag=1.1.0

   ── expected output ──

   # aws_ecs_service.api will be updated in-place

   ~ resource "aws_ecs_service" "api" {

   ~ task_definition = "arn:aws:ecs:...:task-definition/shoplite-dev-api:1" -> (known after apply)

   # (15 unchanged attributes hidden)

   }

    

   # aws_ecs_task_definition.api must be replaced

   -/+ resource "aws_ecs_task_definition" "api" {

   ~ container_definitions = jsonencode(

   ~ [

   ~ {

   ~ environment = [

   ~ { name = "APP_VERSION", value = "1.0.0" -> "1.1.0" },

   # (1 unchanged element hidden)

   ]

   ~ image = "...shoplite/api:1.0.0" -> "...shoplite/api:1.1.0"

   # (6 unchanged attributes hidden)

   },

   ]

   ) # forces replacement

   ~ revision = 1 -> (known after apply)

   }

    

   Plan: 1 to add, 1 to change, 1 to destroy.

   ...

   Apply complete! Resources: 1 added, 1 changed, 1 destroyed.

Make each change, run the command, and read the error before revealing the diagnosis. Recognising these messages on sight is what makes you fast on a real team. Undo the change afterwards.