Designing a Bazel Remote Cache‑First Setup for Scalable CI

Requirements

When scaling continuous integration, teams typically need three guarantees: builds that are reproducible and hermetic, fast incremental feedback on code changes, and lower per‑developer compute cost. Bazel satisfies these by sandboxing actions, using content‑addressed identifiers for inputs and outputs, and exposing a Remote Execution API that separates build planning from execution.

The Smallest Suitable Design

Start with a read‑write remote cache while keeping execution local. This gives most of the benefit of cache reuse without adding the complexity of remote executors. The steps are:

1. Configure a hermetic workspace: declare all tools and libraries in WORKSPACE or MODULE.bazel files, avoid reliance on host‑installed binaries.
2. Enable the remote cache for action results: set --remote_cache=http://cache.example.com and --remote_upload_local_results=true.
3. Keep execution local: do not set --remote_executor; let Bazel run actions on the CI agent.
4. Mark any action that could read secrets or non‑deterministic inputs as nocache or isolate it to a local‑only target.

Only add remote execution later if measured build times or heterogeneous toolchains justify the extra operational overhead.

Trust and Data Boundaries

The build graph (the set of declared dependencies) is trusted; Bazel treats it as the source of truth for what may be cached. The remote cache stores only content‑addressed blobs, so even if the cache service is compromised, an attacker cannot forge a valid output without knowing the correct input digest.

Secrets must never appear as inputs to cached actions. If a target needs a secret (e.g., a signing key), declare it as a --define or pass it via an environment variable that is not propagated to the action’s sandbox, or annotate the target with nocache.

The execution sandbox isolates the action from the host filesystem and network unless you explicitly allow access via platform constraints (e.g., --host_sandbox). Keeping these relaxations minimal preserves hermeticity and makes remote execution portable.

Operational Checks

Monitor the following metrics to verify that the design is working as intended:

• Cache hit rate – proportion of actions served from the remote cache versus executed locally. Aim for a steady increase after warm‑up.
• Action cache misses and upload/download errors** – spikes indicate network issues or mis‑configured cache endpoints.
• Test flakiness under sandbox** – run the same test twice in a clean sandbox; differing outcomes point to non‑hermetic inputs.
• Build event stream** – inspect build_event_text_proto for fields like queue_time, execution_time, and cache_download_failed.

To validate hermeticity, run a representative build on two freshly provisioned CI agents (no shared caches) and compare the output digests of key artifacts (e.g., //:my_binary). Identical digests confirm that the build is independent of the host environment.

Failure Modes

Even with a sound design, certain conditions can degrade reliability:

• Remote cache unavailability** – builds fall back to local execution, increasing CI latency. Design the cache endpoint with high availability and configure a short timeout so the fallback is predictable.
• Non‑hermetic actions** – if an action reads a file outside its declared inputs (e.g., /etc/hosts), the cache may return stale results, breaking reproducibility. Auditing BUILD files for srcs that use globs or package attributes helps catch these.
• Platform‑specific toolchain mismatches** – remote execution (if later added) requires the executor to have the same toolchain version as the planner. Mismatches cause action execution failures that are not visible in the cache.
• Cache poisoning** – content addressing prevents forged outputs, but a compromised cache service could serve incorrect blobs if the integrity check is disabled. Always keep --remote_accept_cached enabled and verify the cache service uses TLS and authentication.

Conditions That Would Change the Design

Revisit the architecture when any of the following occurs:

1. Measured local execution time becomes a bottleneck (>30 % of total CI cycle) and adding remote executors reduces queue time significantly.
2. The development team adopts heterogeneous operating systems or CPU architectures, making a single local executor insufficient for cross‑platform builds.
3. Security policy requires that build outputs be generated in a trusted, isolated environment (e.g., a dedicated VM) rather than on shared CI agents.
4. Cache storage costs exceed budget, prompting a review of garbage‑collection policies or a shift to a tiered cache (hot SSD, cold object storage).

Practical Example Configuration

The following snippet shows a minimal .bazelrc for a CI server that uses a remote cache but runs actions locally:

# .bazelrc
# Remote cache endpoint (replace with your actual URL)
build --remote_cache=https://cache.example.com
# Upload results of locally executed actions to the cache
build --remote_upload_local_results=true
# Do not download cached results if the cache is unreachable; fail fast
build --remote_timeout=30s
# Ensure sandboxing is on (default, but explicit for clarity)
build --sandbox_host_access=false
# Mark any target that uses a signing key as nocache
test --nocache_test_targets=//:sign_release

To verify cache reuse, run a build and inspect the build event output:

bazel build //:my_binary --build_event_text_file=/tmp/bep.txt
# Then grep for cache hits:
grep 'cache_hit' /tmp/bep.txt | wc -l

If the count rises after the first warm‑up run, the remote cache is being used. To test reproducibility, execute the same target on two clean agents and compare the SHA‑256 of the output:

# Agent A
bazel build //:my_binary
sha256sum bazel-bin/my_binary > /tmp/digest_a
# Agent B (clean environment)
bazel build //:my_binary
sha256sum bazel-bin/my_binary > /tmp/digest_b
diff /tmp/digest_a /tmp/digest_b
# No output indicates identical digests.

These checks give concrete evidence that the remote cache is delivering the expected benefits without sacrificing hermeticity.