amazon-ebs builder force_deregister flag: handling AMIs attached to running instances

Confirmed facts

When force_deregister = true is set in the amazon-ebs builder, Packer performs the following steps before creating a new AMI:

1. Searches for an existing AMI with the exact name tag (or names filter) in the target region.
2. If found, calls the AWS DeregisterImage API on that AMI ID.
3. Proceeds with the build (snapshot creation, registration of the new AMI) regardless of the outcome of the deregistration call.

AWS permits DeregisterImage on an AMI even when one or more EC2 instances are currently running from that image. The deregistration is a control‑plane metadata operation; it does not stop, reboot, or otherwise affect the running instances. Those instances continue to use their existing root volume and remain operational until they are terminated or the volume is detached.

Likely explanation for CI vs. production differences

The observed inconsistency is most likely due to one of the following:

• IAM permission variance – The CI role may lack ec2:DeregisterImage for the specific AMI ARN or for * in the region, causing the deregistration call to return an UnauthorizedOperation error that Packer treats as a failure.
• Eventual consistency delay – After a previous build deregisters an AMI, the AWS control plane may take a few seconds to propagate the change. If the next Packer run starts immediately (as can happen in a tight CI loop), the search for the AMI may still find it, leading to a second deregistration attempt that fails because the image is already in the process of being removed.

Steps to resolve or diagnose the issue

1. Verify that the IAM policy used by both CI and production includes ec2:DeregisterImage on the target resource (e.g., "arn:aws:ec2:::image/*") or on * for the region.
2. Enable Packer debug logging (PACKER_LOG=1) and capture the exact AWS API error code returned during the deregistration phase.
3. Check AWS CloudTrail for DeregisterImage events around the time of the build to see whether the call succeeded, was denied, or was throttled.

If the IAM policy is correct and the CloudTrail logs show a successful deregistration, the remaining variable is likely propagation delay. In that case, adding a short retry or wait (e.g., a sleep 10 step or using Packer’s wait_timeout on the builder) before the deregistration search can eliminate the race condition.

One missing diagnostic detail that would change the recommendation

Do you see any DeregisterImage errors (such as UnauthorizedOperation or InvalidAMIID.NotFound) in the Packer build logs or CloudTrail for the failing runs? Knowing the specific error code will tell us whether the problem is permission‑related or a consistency/timing issue, which determines whether to fix IAM policies or add a retry/wait.