Question
Terraform ELB Target Group Migration with create_before_destroy: Controlling Traffic Shift Order
Sky Indigo
0 reputation · 22 Jun 2020, 08:50 UTC
117.1K views1
Loading…
0 reputation · 22 Jun 2020, 08:50 UTC
The goal is to migrate a small application behind an AWS Elastic Load Balancer (ELB) target group without downtime by using Terraform's lifecycle { create_before_destroy = true } to provision a replacement target group before destroying the original.
However, Terraform does not control the order in which the ELB registers targets or deregisters the old group, and the health-check timing is external to the plan. This creates uncertainty about whether the new target group will be healthy and receiving traffic before the old one is drained, which could cause a brief request gap.
We need to decide whether to rely on Terraform's depends_on to enforce ordering, to adjust ELB health-check grace periods, or to introduce external automation (e.g., AWS CodeDeploy, Lambda) that verifies the new target group's health before allowing the destroy step.
Should we add an explicit depends_on on the ELB listener rule or target group attachment to guarantee the new group is created first?
Is adjusting the ELB health-check interval and timeout sufficient to ensure the new group is considered healthy before traffic is shifted?
Do we require external tooling to poll the target group health and only then proceed with the destroy step?
31275 reputation · 22 Jun 2020, 15:05 UTC
To guarantee a zero-downtime migration when using create_before_destroy = true, you cannot rely on depends_on alone to manage traffic flow. Terraform manages the existence of resources, but the AWS ELB manages the state of traffic. An explicit depends_on ensures the new Target Group exists, but it does not ensure the targets within that group have passed their health checks before the Listener Rule shifts traffic.
Confirmed: Terraform will create the new Target Group, update the Listener Rule to point to the new group, and then destroy the old group. The create_before_destroy lifecycle hook prevents the old group from being deleted before the new one is provisioned.
Likely Gap: There is a high probability of a request gap if the Listener Rule is updated immediately after the Target Group is created. The ELB will begin routing traffic to the new group's targets while they are still in the initial health state, leading to 502 or 503 errors until the first successful health check cycle completes.
terraformname_prefix rather than a static name to avoid naming collisions during the create_before_destroy cycle.deregistration_delay.timeout_seconds on the Target Group. This ensures the old group continues to serve existing connections while the new group ramps up.health_check settings (interval and healthy threshold) to be as aggressive as your application allows to minimize the window between registration and healthy status.healthy should you trigger the Terraform apply that updates the Listener Rule.To verify the sequence of operations before applying changes, use the plan command to ensure the new resource is created before the old one is destroyed:
terraform plan | grep -E "create|destroy"To verify the actual traffic drain period in AWS, check the deregistration delay setting:
aws elbv2 describe-target-groups --target-group-arns <tg-arn> --query 'TargetGroups[0].DeregistrationDelay'Are you using Auto Scaling Groups (ASG) to manage the targets? If so, the shift order is further complicated by the ASG's own instance refresh policies, which may require a different orchestration strategy than standalone target registration.
Use comments to ask for clarification. Post a solution as an answer.
31,275 reputation · 22 Jun 2020, 17:20 UTC
To clarify the limitation of depends_on in this scenario: Terraform's dependency graph only tracks the API response from AWS. When aws_lb_target_group is created, AWS returns a success code as soon as the resource exists, not when the targets within it have passed their first health check.
If you want to avoid external Lambda polling, a practical internal verification step is to monitor the HealthyHostCount metric in CloudWatch during your first test migration. You will likely observe that the aws_lb_listener_rule updates almost immediately after the target group is created, meaning traffic shifts while targets are still in the initial state.
To mitigate this without external tooling, ensure your deregistration_delay.timeout_seconds is set significantly higher than your health_check.interval multiplied by the healthy_threshold. This allows the old group to continue draining connections while the new group completes its asynchronous health-check cycle.