Turning Cold CI Builds Warm with Bazel Remote Caching

The cold-start tax on every CI run

Your CI pipeline runs on ephemeral runners. Every commit triggers a fresh container, an empty workspace, and a full rebuild of every target. Even when only one file changed, Bazel re-executes compilation, linking, and codegen for the entire dependency graph. The compute cost is predictable; the frustration is not.

Remote caching removes that tax. By sharing build outputs across machines, Bazel can skip re-execution entirely when an identical action has already run elsewhere. The result is a warm build instead of a cold one. But the cache is only as correct as your build graph, and that is where most teams stumble.

Action-level caching, not output hashing

Bazel’s remote cache is content-addressed at the action level. An action is a single command line plus its declared inputs and environment. If those three things are identical to a previous run, Bazel downloads the cached output instead of running the command again.

This matters because it is finer-grained than hashing final artifacts. A cache hit on a CppCompile action means the compiler never launches. A hit on a linking action means the linker never launches. The cache key incorporates the exact toolchain, the exact flags, and the exact input files—so changing an unrelated target does not invalidate the key for the target you care about.

A worked example: from disk cache to remote cache

The simplest way to understand remote caching is to simulate it locally with a disk cache. On your CI runner or local machine, run:

bazel clean
bazel build --disk_cache=/tmp/bazel-cache //...

Run the same command a second time. Bazel reports a high cache-hit ratio because the second run finds the outputs from the first run on disk. The --disk_cache flag requires write permission to the specified directory and no network access. The flags and protocol behavior described here assume a recent Bazel release; consult the documentation for your version if behavior differs.

To move to a remote cache, point Bazel at an HTTP server that speaks the Bazel Remote Cache protocol. Use https://cache.example.com with authentication credentials; use http://localhost:8080 only for local testing. The CI runner requires network access to the cache host and permission to read the configured credentials:

bazel build --remote_cache=https://cache.example.com //...

After the first build populates the cache, subsequent builds on any machine with access to that URL will fetch artifacts instead of executing actions.

To verify the cache is working, run bazel clean, then build twice with --remote_cache=http://localhost:8080. The second run should show a high cache-hit ratio in the build event summary. You can also inspect the cache server logs or storage directory to confirm artifacts are uploaded after the first build and downloaded on subsequent runs.

The hermeticity contract

Remote caching does not fix non-hermetic builds; it amplifies their breakage across machines. Hermeticity means an action’s output depends only on its declared inputs, command line, and environment. If a rule reads an undeclared file, embeds an absolute path, or depends on the current timestamp, the cache key will not capture that dependency.

The failure mode is subtle. A developer machine populates the cache with an artifact built from a stale or undeclared input. CI downloads that artifact and fails—or worse, succeeds with incorrect output. The cache did not cause the bug; it distributed it.

You can test your hermeticity with bazel aquery. To confirm that action keys do not change when unrelated targets are modified, run:

bazel aquery 'mnemonic("CppCompile", //...)'

Compare the action keys before and after an unrelated edit. If they shift, something in the action’s environment or inputs is leaking global state. For a harder test, temporarily add an undeclared file read to a rule and rebuild. A correct cache will miss, proving the action key includes the new input.

Security and operational limits

Cache entries are executable artifacts. An unauthenticated or plaintext cache is a supply-chain risk: a tampered artifact served to one build compromises every downstream consumer. Use TLS, authentication, and ideally a private network or authenticated proxy.

Storage grows with build diversity. Every unique toolchain, flag combination, and input change creates new entries. Plan retention policies and eviction before the disk fills. Also remember that cache misses on remote execution still require upload bandwidth; large outputs can saturate CI network links.

Rollout: measure before you scale

Enable the cache in CI first, before developer workstations. Measure hit rates with build event streams. If hit rates are low, the problem is usually non-hermetic actions or excessive build diversity, not the cache itself. Once CI hit rates are stable and high, extend the same --remote_cache flag to developer machines so local builds benefit from artifacts already produced in CI.

Remote caching is not a flag you flip; it is a contract. Keep your actions hermetic, protect the cache endpoint, and monitor storage. Do that, and your CI runners stop rebuilding what already exists.