Configure Bazel Remote Caching for Faster CI Builds

Desired outcome

Set up Bazel to store and retrieve build outputs from a remote cache so that CI agents can reuse previously built artifacts, reducing build times and avoiding redundant work.

Prerequisites

• Bazel version 0.23 or newer (remote cache support was stabilized in this release).
• Network access from the CI agents to the remote cache endpoint (gRPC or HTTP).
• Credentials or token if the remote cache requires authentication (e.g., API key, mTLS).
• A writable directory on each agent for the local Bazel output base (--output_user_root) to store temporary logs.

Procedure

1. Choose a remote cache implementation

   Options include a self‑hosted bazel-remote service, a cloud storage bucket with the grpc-web adapter, or a managed service such as BuildBuddy or EngFlow. Ensure the service supports the Bazel Remote Cache API (gRPC).

2. Create a minimal .bazelrc snippet

   Add the following lines to a file that is checked into the repository (e.g., ci.bazelrc) or to each agent’s local .bazelrc:

   # ci.bazelrc
   build --remote_cache=grpcs://:
   build --remote_upload_local_results=true
   # Optional: enable credentials if using mTLS or token
   build --remote_header=x-bazel-credential=
   

3. Verify the cache is reachable

   Run a simple test build that will produce a cacheable output (e.g., a Java library). The first execution will populate the cache; the second should hit it.

   # First run (populate cache)
   bazel build --config=ci //my/project:lib
   # Second run (expect cache hit)
   bazel build --config=ci //my/project:lib
   
   Look for lines containing Remote cache hit in the command output or in the events.json log (bazel info --json shows cache_hit counters).

4. Ensure hermeticity to avoid false misses

   Remote cache keys are based on input file contents, timestamps, and the action environment. Prevent host‑specific influences:

   • Avoid using $(pwd) or absolute paths in srcs; prefer //my/project:file labels.
   • Do not read environment variables that are not declared in --action_env or --host_action_env.
   • Enable sandboxing (default in recent Bazel) to isolate actions from the host filesystem.

5. Optional: Pin a specific Bazel version

   If you upgrade Bazel, remote cache compatibility is generally preserved, but major releases may change the action key format. To be safe, lock the Bazel version in your CI configuration (e.g., use_bazel_version = 6.5.0 in .bazelversion).

Expected checks

• After the warm‑up build, subsequent builds should show Remote cache hit for the majority of actions.
• Run bazel info --json and verify that cache_hit_count increases while cache_miss_count stays low.
• Inspect the remote cache storage (if you control it) to confirm that content‑addressed blobs are being uploaded (grpc traffic or object count growth).

Recovery options

• If the remote cache becomes unreachable, Bazel will automatically fall back to local execution because the --remote_cache flag is advisory. Ensure that --remote_upload_local_results=true is set so that successful local builds still populate the cache when connectivity is restored.
• To force a local rebuild and bypass the cache temporarily, add --noremote_cache to the command line.
• In case of corrupted cache entries (e.g., due to a mismatched toolchain), you can wipe the remote cache for a specific prefix: most implementations support a delete‑by‑prefix API, or you can change the cache namespace by adjusting --remote_instance_name.

Limitations and practical verification

Remote caching does not eliminate the need for correct hermetic builds. If an action reads a file outside its declared inputs (for example, a hard‑coded path to /etc/hosts), the cache key will not capture that dependency and you may get stale outputs. To detect such issues:

1. Run the build with --experimental_action_listener=//tools:cache_audit_listener (a simple listener that logs all accessed files).
2. Compare the file list from two runs with identical sources; any extra file indicates a hidden dependency.
3. Fix the BUILD file to declare the file as an input or move it inside the sandbox.

Finally, always verify that the build outputs are functionally identical after a cache hit by running your test suite (bazel test //...) on both a warm cache and a cold cache; the results should match.