Integrating Bazel Remote Cache with an On‑Premise Build Farm: A Minimal Architecture

Problem Statement

Teams that use Bazel often face two intertwined challenges: 1) repeated builds on many machines waste disk space and bandwidth, and 2) local build farms can become saturated when many developers or CI jobs run concurrently. Bazel’s remote caching and execution features address these issues, but they introduce new trust and consistency concerns. This note lays out a minimal, production‑ready architecture that balances performance, security, and operational simplicity.

Requirements

• All build agents must run the same Bazel version (e.g., 6.5.0) and use identical toolchains.
• Network connectivity between agents, cache, and executor must support gRPC on the chosen ports.
• Authentication for cache and executor traffic (TLS + token or mutual TLS).
• Monitoring of cache hit/miss ratios and executor latency.
• Fallback to local cache when the remote cache is unavailable.

Minimal Architecture

The simplest viable design decouples caching from execution. A local in‑memory cache is retained for each agent, while a central content‑addressable store (CAS) holds shared artifacts. Remote workers are provisioned behind a load balancer that routes gRPC requests to a pool of homogeneous machines.

Diagram of the flow:

Developer Agent
  |  --remote_cache=grpc://cache.example.com:8980
  |  --remote_executor=grpc://executor.example.com:9090
  v
+-----------------------------------+
|          Remote Cache (CAS)       |
+-----------------------------------+
          |
+-----------------------------------+
|      Remote Workers (Toolchain)  |
+-----------------------------------+

Configuration example for .bazelrc on each agent:

# .bazelrc
build --remote_cache=grpc://cache.example.com:8980
build --remote_executor=grpc://executor.example.com:9090
build --experimental_remote_execution
build --experimental_remote_upload
build --experimental_remote_download
build --experimental_remote_cache
build --experimental_remote_cache_timeout=30
build --experimental_remote_executor_timeout=120
build --remote_header=Authorization:Bearer <token>
build --remote_header=Content-Type:application/x-protobuf
# Optional: enable local cache fallback
build --experimental_remote_cache_fallback

Replace <token> with a short‑lived bearer token issued by your authentication service. Ensure the cache and executor endpoints are reachable from all agents.

Trust and Data Boundaries

Remote caches can be a vector for cache poisoning if untrusted data is accepted. To mitigate this:

• Enable --experimental_remote_cache_authentication on the cache server and enforce token validation.
• Sign cache entries with a per‑environment key if your organization supports it. Bazel currently supports --experimental_remote_cache_signature for this purpose.
• Run the cache and executor on isolated networks; restrict inbound traffic to the dedicated gRPC ports.
• Audit cache content periodically: download a known artifact and verify its hash against the expected value.

Operational Checks

To verify that the remote cache and executor are functioning as expected, run the following diagnostics from a developer machine:

1. Cache Hit Ratio Check

   bazel build //app:binary --remote_cache=grpc://cache.example.com:8980
   # Inspect the build output: look for "% cache hit %" line
   # Example output fragment: % cache hit % 70.0%
   

   Expectation: a non‑zero hit ratio indicates that the cache is being used.

2. Remote Executor Activity

   bazel build //app:binary --remote_executor=grpc://executor.example.com:9090
   # In the worker logs, you should see entries like "Executing action: "
   

   Expectation: actions are dispatched to the executor and completions are logged.

3. Cache Integrity Test

   # Modify a source file
   sed -i 's/old_text/new_text/' src/main.cpp
   bazel clean
   bazel build //app:binary
   # Expect a cache miss for the modified action
   # Restore the file
   sed -i 's/new_text/old_text/' src/main.cpp
   bazel build //app:binary
   # Expect a cache hit after restoration
   

   Expectation: the build system detects the change, misses the cache, then restores the original hash and hits the cache again.

Failure Modes and Mitigations

• Cache Poisoning – Mitigated by authentication and optional signature verification.
• Non‑Deterministic Builds – Ensure all workers run the same Bazel version and toolchain; use bazel --version and bazel query --toolchain_resolution=direct to verify consistency.
• Network Latency – Deploy workers in the same data center or use a CDN‑like edge cache for the CAS. Monitor round‑trip times with bazel build --experimental_remote_executor_latency_log.
• Worker Heterogeneity – Use containerized workers (e.g., Docker images pinned to a specific Bazel version) to guarantee identical environments.
• Service Outage – The --experimental_remote_cache_fallback flag ensures that if the remote cache is unreachable, the build falls back to the local cache, preventing a build failure.

When to Re‑evaluate the Design

Consider revisiting this architecture under the following conditions:

• The cache hit ratio drops below 30% for a sustained period, indicating that local builds are not benefiting from the shared cache.
• Build times increase on the executor side due to high latency; this may require relocating workers closer to the agents or adding more workers.
• Security audits reveal that the authentication mechanism is insufficient; consider moving to mutual TLS or integrating with your organization’s SSO.
• The organization adopts multiple toolchains (e.g., C++ and Rust) that cannot share a single CAS; you may need to partition the cache or use separate executors per toolchain.

By keeping the architecture simple—decoupled cache and execution, authenticated traffic, and clear monitoring—you can scale a Bazel build farm while maintaining trust and consistency.