Diagnosing and Fixing Bazel Action Cache Misses

The Problem: Unexpected Rebuilds

A Bazel build should be deterministic: if the inputs haven't changed, the output should be retrieved from the cache. When Bazel rebuilds a target despite no source changes, it is usually due to an Action Cache Miss. This happens when the action's input digest—a hash of all files, environment variables, and command-line flags—changes between runs, forcing Bazel to re-execute the action.

Identifying the Cause

Before attempting fixes, identify why the cache is missing. Use the following table to match your symptoms to the likely cause.

Symptom	Likely Cause	Diagnostic Indicator
Builds are slow on first run after a clean, but also rebuild frequently on small changes.	Environment Leakage	Different --action_env values across developer machines.
The same commit produces different binary hashes on two different machines.	Non-deterministic Toolchains	Binary diffs show timestamps or random seeds.
Cache misses occur when moving the project to a different directory.	Absolute Path Embedding	Generated files contain /home/user/workspace/... strings.
A specific custom rule always rebuilds regardless of inputs.	Volatile Inputs	Shell scripts using date or $RANDOM.

Step-by-Step Diagnostic Process

Follow these steps to isolate the specific cause of the cache miss. These steps assume Bazel 6.x or 7.x.

1. Capture Execution Logs:

   Run the build twice. The first run establishes the baseline; the second run should be a cache hit. Use the JSON execution log to compare the two.

   # Run on your local terminal
   bazel build //your/target --execution_log_json_file=log1.json
   bazel build //your/target --execution_log_json_file=log2.json

   Compare log1.json and log2.json. Look for differences in the inputs or environment_variables sections of the failing action.

2. Analyze Binary Differences:

   If the action executes but the output changes (causing downstream misses), use a tool like diffoscope to inspect the binaries.

   # Compare outputs from two clean builds
   diffoscope build_output_1 build_output_2

   If you see differences in timestamps or build IDs, the toolchain is non-deterministic.

3. Verify Action Graph Stability:

   Use bazel query to ensure the dependency graph isn't shifting due to dynamic rule logic.

   bazel query "deps(//your/target)" > deps1.txt
   # Make a non-code change (e.g., edit a comment in a BUILD file)
   bazel query "deps(//your/target)" > deps2.txt
   diff deps1.txt deps2.txt

Applying the Fixes

Fix 1: Stabilizing the Environment

By default, Bazel strips most environment variables. However, using --action_env can introduce instability if the variable is dynamic (e.g., PATH or USER). Instead of passing the whole environment, define specific, static values in your .bazelrc:

# Avoid: --action_env=PATH
# Use: Explicitly set the required variable
bazel build --action_env=JAVA_HOME=/usr/lib/jvm/java-17-openjdk

Fix 2: Enforcing Toolchain Determinism

Compilers often embed timestamps or random seeds. For GCC or Clang, use flags to strip this data. Add these to your copts in the cc_library or cc_binary rule:

• -frandom-seed=123: Ensures the compiler uses a fixed seed for optimizations.
• -Wno-builtin-macro-redefined: Prevents warnings from shifting based on environment.

Fix 3: Removing Absolute Paths

If custom rules generate files, ensure they use relative paths. If you must use absolute paths, use the root_path provided by the Bazel sandbox. In shell scripts, avoid pwd; instead, rely on the relative paths Bazel provides in the execution root.

Verification and Limitations

To verify the fix, run the build, perform a bazel clean, and run the build again. The second run should report (cached) for the previously problematic targets.

Limitations:

• Performance Trade-offs: Disabling certain compiler optimizations to achieve determinism may slightly impact runtime performance.
• OS Variance: Sandbox behavior differs between Linux and macOS. A build that is deterministic on Linux may still leak environment variables on macOS due to how the OS handles process spawning.

Rollback Procedure

If the changes to .bazelrc or toolchain flags cause build failures or performance regressions:

1. Revert the --action_env changes in .bazelrc.
2. Remove the -frandom-seed or path-stripping flags from the BUILD files.
3. Run bazel clean --expunge to clear the corrupted action cache and start from a known state.