Diagnosing Bazel Action Cache Misses and Unexpected Rebuilds

The Problem: The 'Ghost' Rebuild

Bazel is designed for hermeticity, meaning a build should produce the exact same output given the same inputs. However, developers often encounter "ghost rebuilds": scenarios where no source code has changed, yet Bazel re-executes actions, causing build times to spike and defeating the purpose of the Action Cache (AC).

The takeaway: Cache misses are rarely random. They are caused by non-determinism—where an input, environment variable, or toolchain behavior changes the action's fingerprint (the hash Bazel uses to identify the task) or the resulting output binary.

Identifying the Cache Miss

Before attempting a fix, you must determine if the issue is an Action Key mismatch (Bazel thinks the task changed) or Output non-determinism (Bazel thinks the task is the same, but the resulting file is different, causing downstream failures).

Step-by-Step Diagnostic Process

1. Compare Action Fingerprints

To prove a cache miss is happening due to input changes, you need to compare the action fingerprints of two runs that should be identical. Run the following command on your terminal (requires bazel installed and a defined target):

# Run 1: Generate the log
bazel build //your/target:name --execution_log_json_file=log1.json

# Run 2: Generate a second log without changing code
bazel build //your/target:name --execution_log_json_file=log2.json

Analysis: Use a JSON diff tool to compare log1.json and log2.json. Look for the action_key. If the keys differ, Bazel detected a change in the inputs, environment, or command line arguments.

2. Check for Environment Leakage

Bazel attempts to strip the environment, but certain flags can leak volatile system state into the build. Check your .bazelrc or command line for --action_env.

• The Risk: Using --action_env=ALL incorporates your entire shell environment (including PATH, USER, and HOSTNAME) into the action key. If any of these vary between developers or CI runners, the cache is invalidated.
• The Fix: Explicitly define only the required variables. Replace --action_env=ALL with specific needs: --action_env=JAVA_HOME=/opt/jdk.

3. Detect Binary Non-Determinism

If the action key is the same but the output file changes, the toolchain is non-deterministic. This often happens when compilers embed absolute paths or timestamps into the binary.

Run the build on two different machines (or in two different directories) and compare the outputs:

# Run on Machine A
bazel build //your/target:name
cp bazel-bin/your/target/output_bin output_a

# Run on Machine B
bazel build //your/target:name
cp bazel-bin/your/target/output_bin output_b

# Compare the binaries
diff output_a output_b

Analysis: If diff reports differences, use a hex editor or strings to find the diverging bytes. If you see absolute paths (e.g., /home/user/project/...) or date strings, your toolchain is breaking hermeticity.

Fixes Based on Findings

Finding: Absolute Paths in Binaries

This usually occurs when a custom rule or a compiler flag (like -g in some versions of GCC/Clang) records the source path. Use path mapping flags to rewrite paths to a relative root, such as -fdebug-prefix-map for C++.

Finding: Missing Input Declarations

If a script reads a file that isn't listed in the srcs or data attributes of the rule, Bazel cannot track it. The build may succeed locally because the file exists on disk, but it will fail or behave inconsistently in a remote cache environment. Ensure every file read by an action is explicitly declared in the rule's inputs.

Finding: Timestamp Instability

Some archive tools (like tar or zip) record the file modification time. Since timestamps differ across clones, the output hash changes. Use flags to force a constant timestamp (e.g., --mtime in some tools) or use a wrapper script to strip timestamps from the output.

Escalation Criteria

If the following conditions are met, the issue likely resides in the Bazel version or a deep-seated toolchain bug rather than a configuration error:

• The action_key is identical, but the output is different, and all environment variables are stripped.
• The issue only persists when using a specific remote execution platform (e.g., RBE) despite identical local toolchains.
• The --sandbox_debug output shows the action accessing files outside the sandbox that are not declared as inputs.

Verification and Rollback

Verification: To verify the fix, run the build twice with --execution_log_json_file. The action_key for the target must be identical across both runs, and the resulting binary should be byte-for-byte identical (verified via sha256sum).

Rollback: If changes to .bazelrc (like removing --action_env=ALL) cause build failures due to missing dependencies, revert the .bazelrc change and explicitly add the missing environment variables one by one.