Diagnosing Bazel Cache Misses and Non-Hermetic Rebuilds

The Problem: Unexpected Rebuilds

A primary benefit of Bazel is its ability to avoid redundant work via the action cache. However, when a build triggers recompilation despite no changes to source code or dependencies, the build is "non-hermetic." This means the action key—the unique fingerprint Bazel uses to determine if an action needs to run—is changing between executions.

The immediate takeaway: Do not rely on bazel clean to fix this. Cleaning the cache removes the symptoms but does not identify why the cache is being invalidated, ensuring the problem persists in your CI/CD pipeline.

Identifying the Cache Invalidation Trigger

Before attempting fixes, you must determine why Bazel believes the action is new. Use the following table to match your symptoms to the likely cause.

Symptom	Likely Cause	Diagnostic Signal
Builds are cached locally but miss in CI	Absolute path leakage	Action fingerprints differ by workspace root path
Intermittent rebuilds on different workers	Undeclared environment variables	Differing PATH or HOME values in action logs
Every build triggers a full re-run	Non-deterministic rule outputs	Timestamps or now() calls in custom rules
"Works on my machine" but fails in CI	System toolchain dependency	Reliance on /usr/bin/gcc instead of a hermetic toolchain

Step-by-Step Diagnostic Process

1. Capture Action Fingerprints

To prove a cache miss is happening, you need to compare the execution logs of two builds that should be identical. Run the build twice on the same machine using the --execution_log_json_file flag.

# Run 1: Initial build
# Permissions: Standard user
# Risk: Generates a large JSON file for complex builds
bazel build //my/target --execution_log_json_file=log1.json

# Run 2: No-op build (should be cached)
bazel build //my/target --execution_log_json_file=log2.json

Compare the two JSON files. Look for the action_key. If the keys differ, the inputs to the action have changed. If the keys are the same but the action ran anyway, the issue is likely related to the remote cache configuration rather than hermeticity.

2. Check for Absolute Path Leakage

Bazel expects paths to be relative to the execution root. If a compiler flag or a generated header contains an absolute path (e.g., /home/user/project/main.c), the action key will change whenever the project is moved or run on a different machine.

Verification: Inspect the command_line field in the execution log. Search for strings starting with /home or /Users. If found, you must modify the build rule to use relative paths or $(location) expansions.

3. Audit Environment Variables

Bazel strips most environment variables by default to ensure hermeticity. However, if a rule explicitly requests an environment variable that varies across machines, it will invalidate the cache.

Check: Search your BUILD files for use_default_shell_env = True or custom env attributes in genrule. If a tool relies on an undeclared variable, it may produce different output binaries, leading to downstream cache misses.

4. Analyze Custom Rule Determinism

Custom Starlark rules that embed the current time or a random seed into a file will always produce a new output. Because the output file's hash changes, every action that depends on that file will also rebuild.

Verification: Use bazel dump --skyframe to see the dependency graph. Identify the node that is constantly changing. If a genrule is the source, check if it uses date or hostname commands.

Fixes and Resolutions

• For Path Leakage: Replace absolute paths with $(location target) in genrule or use ctx.label.package in custom rules to maintain relativity.
• For Toolchain Drift: Move from local toolchains to a hermetic toolchain (e.g., a downloaded LLVM toolchain) defined in your WORKSPACE or MODULE.bazel file. This ensures the exact same compiler binary is used regardless of the host OS.
• For Non-Deterministic Outputs: Remove timestamping from generated files. If a version string is required, derive it from the Git commit hash (passed as a build setting) rather than the current system time.

Escalation Criteria

If the following conditions are met, the issue is likely a Bazel bug or a deep filesystem interaction rather than a rule configuration error:

• Execution logs show identical action_key values, but the action is still executed.
• The issue only occurs when using a specific remote cache implementation (e.g., gRPC cache) but disappears with a local disk cache.
• The build fails to cache even when --sandbox_debug is enabled and all inputs are strictly defined.

Rollback and Verification

Since these diagnostics are read-only, there is no state to roll back. To verify a fix, run the build, modify a non-related file, and ensure the target in question remains (cached). Finally, run the build on a second machine or in a clean Docker container to confirm the action key remains stable across environments.