Speeding Up Bazel CI Builds with Remote Caching

Problem: Repeated work in CI pipelines

When a continuous‑integration system rebuilds the same target on every commit, Bazel often repeats compilation and test actions that have already been executed on a developer’s machine. Even with local caching, each CI agent starts with an empty cache, leading to longer pipeline times and wasted compute.

Thesis: Enabling a shared remote cache lets Bazel reuse action outputs across machines, cutting incremental build time while requiring careful attention to hermeticity and cache hygiene.

How remote caching works

Bazel treats each build step as an action with inputs, a command line, and an environment. The action’s output hash serves as a cache key. When --remote_cache is set, Bazel first looks for the hash in the remote store; if found, it downloads the outputs instead of re‑running the action. On a cache miss, Bazel executes the action locally and then uploads the results.

Configuration example

Assume you have an internal HTTP server reachable at https://cache.example.com that accepts PUT and GET requests under the /ac path (the default Bazel remote cache endpoint). Add the following lines to your workspace’s .bazelrc file:

# .bazelrc
build --remote_cache=https://cache.example.com
build --remote_timeout=120
# Optional: use HTTP basic auth if your server requires it
build --remote_header=Authorization=Basic $(echo -n "user:pass" | base64)

Place this file in the root of the repository; Bazel will automatically pick it up for every command. No special permissions are needed beyond the ability to read the file and make outbound HTTPS requests to the cache host.

Worked example: incremental change in a CI job

1. A developer runs bazel build //app:all on their laptop. The first build populates the remote cache with outputs for actions A, B, and C.

2. The CI system picks up the same commit. Before executing any action, Bazel queries the remote cache. Because the inputs have not changed, the hashes for A, B, and C match, so Bazel downloads the cached outputs instead of recompiling.

3. Only the newly modified source file triggers a new action D; Bazel builds D locally and uploads its output to the cache for future use.

To observe the cache in action, you can run a local test with a temporary HTTP server:

# In one terminal, start a simple cache server (requires Python 3)
python3 -m http.server 8080 --directory /tmp/bazel-cache

Then, in another terminal:

bazel clean --expunge
bazel build //app:all --remote_cache=http://localhost:8080
# First run: expect PUT requests in the server logs
bazel build //app:all --remote_cache=http://localhost:8080
# Second run: expect GET requests and no re‑compilation messages for unchanged targets

Check the server’s access log; you should see PUT on the first build and GET on the second. No new compiling lines for unchanged targets indicate a cache hit.

Trade‑offs and limitations

• Network dependency: If the cache server is unreachable or slow, builds may fall back to local execution, but timeout misconfiguration can cause failures.

• Cache correctness: Remote caching assumes hermetic actions. Rules that read environment variables, the filesystem outside declared inputs, or the current time can produce identical hashes for different results, leading to silent mis‑builds.

• Storage management: The cache can grow indefinitely. Implement a garbage‑collection policy (e.g., delete entries older than 30 days) or use a server that supports quota‑based eviction.

• Version skew: Producer and consumer must use compatible Bazel versions and toolchains; otherwise, action hashes may diverge even with identical sources.

Practical verification steps

1. Run a build with --remote_cache pointing to a test server and record the elapsed time (time bazel build ...).

2. Run bazel clean --expunge followed by the same build again; the second build should be noticeably faster if the cache is warm.

3. Inspect the server logs for matching PUT/GET pairs. A high GET‑to‑PUT ratio on repeated builds indicates effective reuse.

4. To guard against non‑hermetic rules, add --noremote_accept_cached temporarily and verify that the build still succeeds; any difference points to a hermeticity issue.

Actionable closing

Start by adding a minimal --remote_cache line to your .bazelrc and point it at an internal HTTP server you control. Monitor the first few CI runs for cache hits via server logs, and adjust timeouts or authentication as needed. Once you see consistent GET requests on unchanged targets, consider enabling cache‑garbage‑collection and documenting the hermeticity requirements for any custom Starlark rules you maintain. This incremental approach lets you reap the speed benefits of remote caching while keeping the risks visible and manageable.