Speeding Monorepo Builds with Bazel Remote Cache and Remote Execution

Problem: Slow Builds in a Large Monorepo

When a codebase grows to hundreds of thousands of lines, a single developer’s rebuild can take minutes, and continuous‑integration (CI) jobs can stall for hours. The culprit is often duplicated work: every worker pulls the same source, compiles the same libraries, and runs identical tests. In Bazel, each build action is a deterministic unit of work, but if the result isn’t reused, the effort is wasted.

Thesis: Remote Cache + Remote Execution Cuts Wall‑Clock Time

Enabling a remote cache stores compiled artifacts in a shared backend (e.g., Google Cloud Storage, Amazon S3, or a Redis instance). Subsequent builds, whether on a developer machine or in CI, can fetch the artifact instead of recomputing it. Adding remote execution offloads heavy actions—such as Docker image builds or large test suites—to a cluster, so the same logical work is done once and reused everywhere.

Setting Up the Remote Cache

The core configuration is a single command‑line flag, but several details matter:

1. Cache URL: The endpoint that Bazel talks to. It must support the GRPC protocol and expose the Bazel cache API.

   # Replace with your actual cache service
   bazel build //:my_app \
     --experimental_remote_cache=grpc://cache.example.com
   

2. Authentication: Most services require a bearer token or a client certificate. Pass the token via a file or environment variable.

   # Example: token in a file
   export BAZEL_REMOTE_CACHE_TOKEN=$(cat ~/.cache_token)
   bazel build //:my_app \
     --experimental_remote_cache=grpc://cache.example.com \
     --experimental_remote_cache_credentials=token
   

3. Local fallback cache: When the network is down, Bazel can still read from a local disk cache.

   # Use a local directory as a fallback
   bazel build //:my_app \
     --experimental_remote_cache=grpc://cache.example.com \
     --experimental_remote_local_cache=/tmp/bazel_local_cache
   

4. Cache size limits: Configure a max size or pruning policy on the backend to avoid runaway storage costs.

Enabling Remote Execution

After the cache is in place, you can add the execution flag. The action that would normally run on the local machine now runs on a remote worker (e.g., Kubernetes, Cloud Build, or a custom cluster). The result is streamed back to the local client.

# Remote execution with the same cache
bazel build //:my_app \
  --experimental_remote_cache=grpc://cache.example.com \
  --experimental_remote_execution=grpc://exec.example.com

When both flags are active, the build engine first checks the cache; if the action key is missing, it sends the work to the remote executor. If the executor completes, the result is written back to the cache for future reuse.

Monitoring and Verifying Cache Effectiveness

• Console output: Bazel prints “Cache hit” or “Cache miss” for each action. Look for a high hit ratio (e.g., >90%).
• Backend metrics: Most storage backends expose metrics like hit count, miss count, and bandwidth usage. Pull these metrics to confirm that the cache is being used.
• Action graph inspection: Run bazel query --output=build to see the action keys and confirm they reference the remote cache URL.

Example verification snippet:

# After a build
bazel build //:my_app
# Check console for Cache hit entries
# Verify storage backend contains the artifact
# (e.g., list objects in GCS bucket)

Trade‑offs and Limitations

• Determinism: Cache hits only occur if the action is deterministic. Non‑deterministic outputs (timestamps, random seeds) cause misses. Add --stamp or --experimental_guard_against_non_determinism to enforce determinism.
• Network latency: If the cache server is geographically distant, round‑trip time can outweigh the benefit. Position the cache close to your developers and CI hosts.
• Cost: Storing large binaries (e.g., prebuilt Docker layers) can inflate storage fees. Enable pruning or set size limits.
• Security: Use TLS and token‑based auth. Restrict cache access to the Bazel service account.

Actionable Next Steps

1. Deploy a lightweight cache service (e.g., bazel‑remote) in your cloud region.
2. Add the --experimental_remote_cache flag to your .bazelrc for local developers and CI jobs.
3. Run a pilot build, capture console output, and compare build times before and after.
4. Monitor cache hit ratios and storage usage. Adjust cache size or pruning policies as needed.
5. Once hit ratios stabilize, add --experimental_remote_execution to offload heavy actions.
6. Document the configuration in your team’s build guide and enforce it via CI lint checks.

By following these steps, you can turn a slow, monolithic build pipeline into a fast, distributed system that re‑uses work across the entire organization.