Speeding Up CI and Local Builds with Bazel Remote Caching

Problem: Rebuilding the Same Code Over and Over

In a large monorepo, a single change can trigger the rebuild of thousands of targets. Even with incremental builds, the first run of a CI job or a local developer’s “make” can take minutes if every action touches the network or writes large artifacts to disk. Bazel’s remote cache promises to reduce that cost by reusing outputs that have already been produced elsewhere. The question is: how do you turn that promise into a measurable speed‑up?

Thesis: Remote Caching Works When Actions Are Truly Hermetic

Bazel stores each action’s output in a cache keyed by a hash of the action’s inputs, the command line, and the environment. If the key matches an existing entry, the output is fetched instead of re‑executed. That works only if the action is deterministic – it must always produce the same result given the same inputs. The rest of this article shows how to configure the cache, verify that it’s being used, and keep the key stable.

1. Setting Up the Remote Cache

Choose a Backend

Bazel supports two transport protocols: gRPC and HTTP. The HTTP protocol is easier to start because it works over plain HTTPS and can be backed by any S3‑compatible store or a self‑hosted cache server. The gRPC protocol offers lower latency but requires a dedicated server.

Configure the Flags

On every machine that should participate in caching (CI agents, developer laptops, build servers), add the following to .bazelrc or pass them as command line options:

# Use the HTTP backend
--remote_cache=http://remote-cache.example.com
# Optional: enable authentication if the backend requires it
--remote_cache_auth_header=Bearer $REMOTE_CACHE_TOKEN
# Keep a local disk cache to avoid round‑trips for hits that are already cached locally
--disk_cache=/var/tmp/bazel-disk-cache
--disk_cache_hardlink=true
# Force cache usage for everything that can be cached
--remote_cache_enable_content_addressable_storage=true

Replace http://remote-cache.example.com with your cache’s URL and set the token environment variable if needed. The --disk_cache flag is optional but highly recommended for developers who run bazel build locally; it reduces network traffic for repeated runs.

Permissions and Network Considerations

• The cache server must allow PUT and GET requests for the object bucket or storage API.
• For large binary artifacts, consider the bandwidth cost. A 100 MB artifact will be uploaded once and downloaded once per cache hit.
• Set a reasonable timeout: --remote_timeout=120 seconds is a good starting point for a gRPC cache; HTTP clients may use --remote_http_timeout=120.

2. Ensuring Cache‑Key Correctness

Hermetic Actions First

Bazel’s cache key includes the hash of every file listed in the action’s inputs and the command line. If an action reads a file that is not declared as an input, or depends on a dynamic environment variable, the key will differ between runs, and the cache will miss.

Use --sandbox_debug to see which files are read by an action. If you see leaks, add them to the inputs or refactor the action to be hermetic.

Control Timestamp‑Based Sources

Generated files that embed timestamps or build dates will always differ, even if the source code is the same. Use genrule or a custom rule that writes a deterministic output, or strip timestamps after generation. Bazel’s --stamp flag can help when you need reproducible builds.

Explicit Dependencies

When a rule uses a tool that is not part of Bazel’s dependency graph (e.g., a system‑wide compiler), the tool’s version can change without Bazel knowing. Make the tool a toolchain dependency so that the tool’s hash becomes part of the action key.

3. Verifying that the Cache is Hit

Run a build with verbose failure output to see cache activity:

bazel build //my/package:target \
  --remote_cache=http://remote-cache.example.com \
  --verbose_failures

Look for the lines:

• remote cache hit – the output was fetched from the remote cache.
• remote cache miss – the action ran locally and its output was uploaded.

Another useful check is bazel aquery --output=textproto //my/package:target. The resulting proto includes a field cache_hit for each action. Counting the number of hits versus total actions gives a hit‑rate metric.

4. Trade‑Offs and Limitations

• Cache Poisoning – If a corrupted or incorrect output is uploaded, every subsequent hit will be wrong until the cache is purged. The safest approach is to purge the entire cache when you upgrade Bazel or change the toolchain.
• Network Latency – For geographically distributed teams, round‑trip time can dominate the perceived speed‑up. In those cases, a per‑region cache or a CDN‑backed object store can help.
• Version Invalidation – The cache key includes the Bazel version and toolchain hashes. A new Bazel release or a compiler upgrade will invalidate many cached actions, leading to a temporary slowdown.
• Non‑Cacheable Actions – Actions that are marked no-cache or that write to the local filesystem outside of the sandbox will always run.

5. Practical Checklist Before You Roll Out

1. Set up a dedicated remote cache server (S3, GCS, or a local cache service) and confirm connectivity from all CI agents.
2. Add --remote_cache and --disk_cache flags to .bazelrc for developers and CI.
3. Run bazel clean --expunge and a full build to populate the cache.
4. Verify cache hits with --verbose_failures and bazel aquery.
5. Measure build times before and after enabling caching; aim for at least a 30 % reduction in total action count for a typical CI run.
6. Document the cache configuration and provide a troubleshooting guide for common miss reasons.

Conclusion: When Remote Caching Pays Off

Remote caching is not a silver bullet, but when the actions in your repository are deterministic and you have a reliable network, the impact on CI duration and local productivity can be substantial. By carefully configuring the cache, ensuring hermeticity, and monitoring hit rates, teams can see consistent build‑time reductions that scale with the size of the codebase. Start small – enable the cache on one CI job, measure the hit‑rate, and iterate. Once the cache is healthy, you’ll notice that both your CI pipeline and your local “build” feel almost instantaneous.