Implementing Bazel Remote Caching to Reduce Distributed Build Times

The Problem: Redundant Compilation in Distributed Teams

In large Bazel projects, developers and CI pipelines often spend significant time recompiling the same code. When a developer pulls the latest changes from a branch that has already been built by CI, Bazel normally rebuilds those targets locally because the local action cache is empty. This redundancy wastes compute resources and slows down the development cycle.

The solution is Remote Caching. By using a gRPC-compatible backend, Bazel can upload build outputs to a shared server. Other clients can then download these pre-computed outputs instead of executing the compilation locally, provided the inputs and toolchains are identical.

Prerequisites for Cache Consistency

Remote caching only works if your builds are hermetic. A build is hermetic when it depends only on explicitly declared inputs and not on the host environment. If your build rules use absolute paths, local timestamps, or system-installed compilers that differ by version between machines, you will encounter cache misses or cache poisoning where an incorrect artifact is shared across the team.

• gRPC Cache Backend: A running instance of a Bazel-compatible cache such as Bazel Remote, BuildBuddy, or a cloud-native storage provider.
• Consistent Toolchains: All clients must use the same compiler versions and system headers. Using a toolchain defined in WORKSPACE or MODULE.bazel is recommended over relying on /usr/bin/gcc.
• Network Connectivity: The client must have gRPC access to the cache endpoint on the specified port.

Configuring the Remote Cache

Configuration is typically handled in the .bazelrc file to ensure consistency across the team. Avoid passing these as command-line flags for every build.

Basic Configuration

Add the following to your .bazelrc to enable the remote cache for all users:

# Define the remote cache endpoint
build --remote_cache=grpc://your-cache-endpoint.com:9092

# Enable the use of the remote cache
build --remote_upload_local_results=false

Note on --remote_upload_local_results: By default, this is set to false. If set to true, every developer's local build will upload results to the shared cache. This is generally discouraged as it can lead to cache pollution if local environments are not perfectly aligned with CI.

CI vs. Developer Permissions

To maintain cache integrity, implement a read-write split. The CI pipeline should be the source of truth that populates the cache, while developers should only consume from it.

For CI Read-Write:

build --remote_cache=grpc://your-cache-endpoint.com:9092
build --remote_upload_local_results=true

For Developers Read-Only:

build --remote_cache=grpc://your-cache-endpoint.com:9092
build --remote_upload_local_results=false

Verification and Diagnostics

To verify that the remote cache is functioning, observe the build logs for remote cache hit indicators.

1. First Run: Execute a build on the CI machine. This will populate the cache.

   bazel build //...

2. Second Run: Execute the same build on a different machine or clear the local cache using bazel clean --expunge.
3. Check Logs: Run the build with the --remote_cache flag active. Look for the summary at the end of the build output.

Expected Result: You should see a significant reduction in execution time and a report indicating that actions were retrieved from the remote cache rather than executed locally.

Troubleshooting Cache Misses

Limitations and Risks

• Network Overhead: If the time to download a large artifact from the remote cache exceeds the time to compile it locally, the cache becomes a bottleneck. This is common in geographically distributed teams without regional cache mirrors.
• Cache Poisoning: If a build rule is incorrectly written to include a local timestamp, the cache key will change every second, causing a 0% hit rate. Conversely, if a rule ignores a critical input, it may serve a stale artifact to other users.

Rollback Procedure

If the remote cache causes build instability or severe latency, disable it by removing the --remote_cache line from .bazelrc or overriding it at the command line:

bazel build --remote_cache= //...

Running the flag with an empty value disables the remote cache for that specific invocation, forcing Bazel to rely solely on the local action cache.