One template with parallel builders or sequential packer builds: which for a latency-sensitive CI runner?
0 reputation · 17 Jun 2025, 10:18 UTC
The trade-off
Packer (HCL2 templates, assuming a recent 1.10+ release) runs every build block in one template concurrently by default, with -parallel-builds available to cap or serialize them. The alternative is splitting images into separate templates and invoking packer build one at a time in the pipeline.
The constraint is a single shared build host with limited CPU cores and disk throughput, where per-build latency feeds a release pipeline but total wall-clock time also matters. Concurrent builders share the cache directory (PACKER_CACHE_DIR) and each spawns its own plugin processes, so concurrency plausibly introduces contention a lone build never shows — but neither the source nor the size of that overhead is established for this setup.
What's unresolved
Before standardizing on either arrangement, the decision hinges on where concurrency-only latency actually comes from:
- Does per-build latency under
-parallel-builds=Ndiverge from a serial run mainly at plugin startup, cache downloads, or host resource saturation? - Can concurrent builds of the same base image contend in the shared cache directory, or are downloads there already serialized?
- What measurement would cleanly separate concurrency overhead from ordinary CPU and disk saturation on the runner?