Packer concurrent builds show latency without per-build resource controls

Direct answer to the question

Packer’s core does not expose per‑build CPU, memory, or I/O quotas. Each build runs as an independent CLI process and inherits whatever limits the host operating system applies. The only built‑in concurrency control is the global -parallelism flag, which caps the total number of builds launched by a single invocation but does not isolate resources per build or provide a queueing mechanism.

Why the latency appears only under concurrency

• Host‑level contention: Multiple builds targeting the same provider compete for CPU cycles, disk I/O bandwidth, and network sockets. This is visible in system metrics such as high CPU wait times, elevated I/O wait, or increased load averages.
• Provider‑side throttling: Cloud APIs impose rate limits. If many builds hit the same endpoint simultaneously, the provider may return 429 (Too Many Requests) or 503 (Service Unavailable) responses. Packer’s retry logic can mask these errors as “just slower” rather than a clear failure.

Observable signals

To differentiate the two, look for:

• System metrics: top, iostat, sar show high CPU wait or disk I/O wait, pointing to host contention.
• Provider logs: 429/503 status codes, increased round‑trip times, or explicit throttling messages in the provider’s SDK logs indicate provider throttling.
• Packer logs: A long Retrying block without an error message often means the provider is throttling, whereas a sudden spike in overall build duration without specific errors points to host resource limits.

Practical steps for this scenario

1. Run a baseline:

   packer build -parallelism 1 template.json
   

   Measure the average build time.
2. Run concurrent builds:

   packer build -parallelism 4 template.json
   

   Compare the total time and note any Packer error codes.
3. Collect host metrics: While the concurrent run is in progress, execute:

   top -b -n 1 | grep -E 'CPU|Load'
   iostat -xz 1 10
   

4. Inspect provider logs: For AWS, look for RequestLimitExceeded or ThrottlingException in CloudTrail or SDK output.
5. Correlate: If system metrics show high I/O wait but provider logs are clean, the bottleneck is the host. If you see provider throttling codes, the delay originates from the cloud provider.

How to mitigate

• Limit concurrency with -parallelism or by launching separate packer build processes with a controlled schedule.
• Run builds in isolated environments (containers, VMs, or cgroups) to enforce per‑process CPU/memory limits.
• Use provider‑specific SDK throttling back‑off or increase quota where possible.

Missing diagnostic detail

To give a more tailored recommendation, could you share which cloud provider you are targeting (e.g., AWS, Azure, GCP) and whether you have any system‑level monitoring data (CPU, disk, network) available during the concurrent runs?