Packer HCL2 and Cloud Provider APIs: Idempotency during Post-Processor Retries

Answer

Amazon‑import does not expose an idempotency token. The API call that registers the AMI is a simple RegisterImage request, which is only idempotent if the Name field is unique. Therefore, a retry of the post‑processor will create a new AMI unless you supply a unique name or otherwise guard the creation step.

Confirmed Facts

• Packer treats each post‑processor as a distinct step and does not retry failed post‑processors automatically.
• The AWS RegisterImage API will reject a duplicate name with InvalidAMIID.Duplicate.
• Adding a unique token (build ID, timestamp, or Git SHA) to the AMI Name makes each retry create a distinct resource.

Likely Explanation

Because the post‑processor has no built‑in retry logic, the only way to avoid duplicate AMIs is to ensure the Name is unique or to skip the registration step if the AMI already exists. Cloud providers generally do not provide an “upsert” operation for AMIs, so the idempotency must be handled at the Packer level.

Practical Steps for Amazon‑import

1. Insert a unique identifier in the AMI name. Use the built‑in build_name or build_id variables, or concatenate a timestamp or Git commit SHA.

   post-processors {
     amazon-import {
       name = "myapp-${build_name}-${timestamp}"   # e.g., myapp-ubuntu-20260919T2319-12345
       ...
     }
   }

2. Use the manifest post‑processor to record the AMI ID. After a successful run, the manifest file contains the artifact_id which you can reference in subsequent runs to skip creation.

   post-processors {
     manifest {
       output_path = "packer-manifest.json"
     }
   }

3. Optional: Pre‑check for existing AMIs. If you prefer to reuse an existing AMI, run a small shell-local script that queries AWS for an AMI with the target name and exits with a non‑zero status if it exists. Packer will skip the amazon‑import step when the script fails.

   post-processors {
     shell-local {
       inline = [
         "aws ec2 describe-images --filters Name=name,Values='myapp-${build_name}' | grep -q 'ImageId' || exit 1"
       ]
     }
     amazon-import {
       ...
     }
   }

4. External retry loop. If you need the entire build to retry on transient failures, wrap the packer build command in a shell loop with exponential back‑off. Do not add the same AMI name on each retry; keep the unique token so that the second attempt creates a new AMI rather than reusing the first.

   for i in {1..3}; do
     packer build -debug mybuild.hcl && break
     sleep $((2**i))
   done

What to Verify

• Run packer build -debug and confirm that the amazon‑import step appears only once per build.
• After a retry, list AMIs with aws ec2 describe-images --filters Name=name,Values='myapp-*' and check that each has a distinct ID.
• If you see InvalidAMIID.Duplicate errors, the name collision is happening and you need to adjust the naming strategy.

Missing Diagnostic Detail

To tailor the recommendation further, could you confirm whether you are using any external retry plugin or script that re‑invokes the amazon‑import step with the same AMI name? This will affect whether the uniqueness strategy above is sufficient.