Cutting CI Build Times with Bazel Remote Caching

The Cost of Redundant Compilation

In large-scale monorepos, the most expensive part of a CI pipeline is often the work that has already been done. When a developer changes a single leaf-node library, a naive CI system might rebuild the entire dependency graph, wasting compute cycles and delaying feedback. While Bazel provides local caching, that cache is wiped every time a CI runner is destroyed, forcing every single build agent to start from scratch.

The solution is Remote Caching. By moving the build artifacts from a local directory to a shared, content-addressable storage (CAS) system, any action performed by one developer or CI worker can be instantly reused by everyone else in the organization.

How the Action Key Determines a Hit

Bazel doesn't cache based on filenames or timestamps; it uses a cryptographic hash called the Action Key. This key is a digest of everything that could possibly affect the output of a command: the command-line flags, the environment variables, the toolchain version, and the digests of all input files.

If the Action Key matches an entry in the remote cache, Bazel simply downloads the resulting artifact instead of executing the tool. This transforms a potentially hour-long compilation into a series of network downloads.

Implementing a Remote Cache Backend

To enable remote caching, you need a gRPC-compatible backend (such as Bazel Remote, BuildBuddy, or Google Cloud Build Cache). Configuration is typically handled in the .bazelrc file to ensure consistency across the team.

Configuration Example

# .bazelrc
# Enable remote caching via gRPC
build --remote_cache=grpc://cache.example.com:9092

# Ensure the cache is used for both reading and writing
build --remote_upload_local_results=true

# Optional: Use a specific cache instance for different branches
# build --remote_instance_name=feature-branch-x

Execution and Verification:
Run the build from your terminal or CI script using the configured flags. To verify the cache is working, run the build twice:

1. First Run: Run bazel build //.... You will see the usual compilation logs. At the end, Bazel will report the number of actions executed.
2. Second Run: Run the same command without changing any code. You should see a significant decrease in time, and the build summary will indicate that actions were retrieved from the remote cache.

The Danger of Cache Poisoning and Non-Determinism

Remote caching is powerful, but it introduces a critical risk: Cache Poisoning. This happens when an action produces different outputs for the same Action Key, and the "wrong" output is uploaded to the shared cache.

• Non-Determinism: If your build script embeds a timestamp or a random seed into a binary, the output changes every time. This causes "flapping," where the cache is constantly overwritten, rendering it useless.
• Toolchain Mismatches: If Developer A uses GCC 11 and Developer B uses GCC 12, but both have the same Action Key (because the toolchain version isn't properly tracked in the Bazel configuration), Developer B might download a binary compiled by Developer A that is incompatible with their system.

Trade-offs: Network Latency vs. Compute

Remote caching is not a silver bullet. There is a physical trade-off between the time it takes to compile locally and the time it takes to download a large artifact over the network. If your remote cache is hosted in a different region than your CI runners, the network latency can actually make the build slower than a local compile for small, fast actions.

To mitigate this, ensure your cache backend is co-located with your compute resources and use a high-bandwidth connection.

Summary Checklist

Before rolling out remote caching to a production team, verify the following:

• Determinism: Ensure no timestamps or absolute paths are baked into binaries.
• Toolchain Locking: Use a hermetic toolchain (e.g., via rules_cc or rules_go) so every user uses the exact same compiler version.
• Network Proximity: Test the download speed of a large artifact from the cache to ensure it is faster than a local rebuild.