BuildKit Registry Cache vs. Local Layer Cache for CI Retry Efficiency

Optimizing Write-Heavy Build Retries

In distributed CI environments using Docker BuildKit, the goal is to prevent the re-execution of expensive RUN instructions that perform heavy filesystem writes when a build is retried or triggered across different agents.

Local layer caching allows a single agent to skip unchanged steps instantly. However, in ephemeral CI runners, this cache is lost between jobs. BuildKit's --cache-from and --cache-to options enable the use of a remote registry backend to share these layers across a cluster, potentially avoiding redundant writes on new agents.

The trade-off involves the overhead of pulling remote cache manifests and layers versus the compute and I/O cost of re-running the write-heavy instructions locally.

Technical Constraints

• BuildKit must be enabled as the active builder.
• Remote cache imports depend on registry authentication and network latency.
• Any modification to a preceding Dockerfile instruction invalidates all subsequent cached layers.

Which approach provides more consistent build times for large-scale filesystem writes in ephemeral environments: relying on local cache persistence via volume mounts or utilizing a registry-based cache backend?

Under what network latency thresholds does the cost of pulling remote layers exceed the cost of re-executing the write instructions?