BuildKit Registry Cache vs. Local Layer Cache for CI Retry Efficiency

Question

Optimizing Write-Heavy Build Retries
In distributed CI environments using Docker BuildKit, the goal is to prevent the re-execution of expensive RUN instructions that perform heavy filesystem writes when a build is retried or triggered across different agents.
Local layer caching allows a single agent to skip unchanged steps instantly. However, in ephemeral CI runners, this cache is lost between jobs. BuildKit's --cache-from and --cache-to options enable the use of a remote registry backend to share these layers across a cluster, potentially avoiding redundant writes on new agents.
The trade-off involves the overhead of pulling remote cache manifests and layers versus the compute and I/O cost of re-running the write-heavy instructions locally.
Technical Constraints

BuildKit must be enabled as the active builder.
Remote cache imports depend on registry authentication and network latency.
Any modification to a preceding Dockerfile instruction invalidates all subsequent cached layers.

Which approach provides more consistent build times for large-scale filesystem writes in ephemeral environments: relying on local cache persistence via volume mounts or utilizing a registry-based cache backend?
Under what network latency thresholds does the cost of pulling remote layers exceed the cost of re-executing the write instructions?

Tasadduq Burney · Answer

Answer
For large‑scale filesystem write steps in ephemeral CI runners, a persistent local layer cache (via a volume mount) yields more consistent build times than a registry‑based cache, because reuse is near‑instant and unaffected by network latency.
Confirmed Facts

BuildKit must be the active builder for either cache method.
Local cache persistence requires the CI runner to provide a storage volume that survives between jobs; otherwise the cache is lost.
Registry cache adds overhead equal to the round‑trip latency plus the time to transfer the cached layers; it only beats local re‑execution when the cost of re‑running the heavy RUN step exceeds that pull/push overhead.
Both methods share the same invalidation rule: any change to a preceding Dockerfile instruction nullifies all later cached layers.

Likely Explanation
When the volume‑backed local cache is available, the cached layer is read directly from disk, giving sub‑second reuse regardless of network conditions. The registry approach incurs variable latency (RTT, bandwidth, possible throttling) and therefore produces a wider spread of build times, especially under congested or high‑latency links.
Steps to Decide for Your Environment

Measure the time to re‑execute the heavy WRITE RUN instruction on a clean agent (no cache). Call this T_reexec.
Measure the time to pull/push the same layer from/to your registry under typical network conditions (include authentication). Call this T_pullpush.
If T_reexec > T_pullpush, the registry cache is expected to be faster; otherwise rely on the local volume cache.
Optional: inject controlled network delay (e.g., tc netem delay 100ms) and re‑measure T_pullpush to see at what latency the recommendation flips.

Missing Diagnostic Detail
If you do not know the actual re‑execution time of the heavy write step on a fresh runner (T_reexec), obtain that measurement first; it directly determines which cache strategy is preferable.

BuildKit Registry Cache vs. Local Layer Cache for CI Retry Efficiency

Optimizing Write-Heavy Build Retries

Technical Constraints

1 answer

Answer

Confirmed Facts

Likely Explanation

Steps to Decide for Your Environment

Missing Diagnostic Detail

Your perspective belongs here.

0 question comments