Accelerate Bazel CI Builds with Remote Caching

Problem

In CI pipelines, Bazel often recompiles the same targets on every run, even when source files have not changed. This wastes compute time and lengthens feedback loops, especially for large monorepos where incremental builds dominate the pipeline.

Takeaway

By configuring a remote cache, Bazel can store and reuse compiled outputs across CI runs. When the cache is warm, subsequent builds hit the cache for unchanged actions, reducing wall‑clock time without changing source code.

Prerequisites

• Bazel version >= 0.28 (remote caching support).
• A reachable remote storage endpoint that Bazel can talk to via HTTP/HTTPS (e.g., Google Cloud Storage bucket, Azure Blob container, or an internal HTTP cache).
• Authentication credentials that allow read and write access to the endpoint. Prefer environment variables or a .netrc file with restrictive permissions (600) over placing secrets directly in .bazelrc.
• Network connectivity from the CI workers to the remote endpoint; measure round‑trip latency to ensure it does not outweigh the benefit for very small builds.

Procedure

1. Add remote cache settings to .bazelrc

   Insert the following lines (adjust the URL and auth method as needed). Using environment variables keeps secrets out of version control.

   # .bazelrc
   build --remote_cache='${REMOTE_CACHE_URL}'
   # Optional: upload local results even if they were not executed remotely
   build --remote_upload_local_results=true
   # If using HTTP basic auth via .netrc, no extra flag is needed.
   # If using token‑based auth, you can set it via environment:
   # build --remote_header='Authorization: Bearer ${REMOTE_CACHE_TOKEN}'
   

   Place the file at the workspace root or in $HOME/.bazelrc for user‑wide settings.

2. Warm the cache

   Run a clean build to populate the remote storage with the initial set of outputs.

   # Ensure a clean state
   bazel clean --expunge
   # Populate the cache
   bazel build //...

   This step may take longer than a normal build because everything is uploaded, but it only needs to happen once per major change in dependencies.

3. Execute regular CI builds

   In subsequent pipeline steps, invoke Bazel as usual. The remote cache will be consulted automatically.

   bazel build //...

   Watch the console output for lines such as:

   Remote cache hit: //some/target:some_action
   

   If you see many Cache hit messages and few Executed messages, the cache is being reused.

Expected Checks

• Count cache hits: after a build, run bazel info --output_base to locate the events.json log, then grep for "Cache hit". Compare the hit count to the total number of actions reported in the same log.
• Measure build time: record the wall‑clock time of a cold build (with --noremote_cache or the remote cache line commented) and a warm build (cache enabled). A noticeable reduction confirms the feature works.
• Verify fallback: temporarily block network access to the remote endpoint (e.g., using a firewall rule) and re‑run the build. Bazel should proceed with local execution and finish without error, albeit slower.

Recovery Options

• If the remote cache becomes unavailable, Bazel automatically falls back to local execution. No manual intervention is required for the build to succeed.
• To disable the cache intentionally (e.g., for debugging), comment out the --remote_cache line in .bazelrc or set --noremote_cache on the command line.
• If authentication fails, check that the environment variables or .netrc file are correctly formatted and have appropriate permissions (600 for .netrc).

Limitations and Practical Tips

• Network latency can dominate for very small builds; run a quick latency test (curl -w "%{time_total}" -o /dev/null -s ${REMOTE_CACHE_URL}) and compare it to the typical local build time.
• Ensure the storage lifecycle policy retains objects long enough for your workflow. Premature deletion causes cache misses and forces rebuilds.
• Avoid storing long‑lived secrets in .bazelrc. Use .netrc or CI‑specific secret injection mechanisms.
• Monitor cache size and egress costs; some providers charge for data transfer out of the bucket.