Bazel remote caching: inconsistent action hashes with identical inputs

Answer

Bazel detects and neutralizes non-deterministic inputs by computing a unique action cache key derived from the command line, the content and paths of input files, declared environment variables, the configuration hash (build flags), and action-specific salts. Any variance in these components—even if the source code is identical—results in a different hash and a cache miss.

Likely Explanation

Hash divergence typically occurs when builds are not strictly hermetic. The most common culprits are differences in host tool versions (e.g., different JDK or Python versions), varying PATH variables, locale settings, or absolute paths leaked into generated files. These environmental differences alter the action's command line or environment manifest, triggering a new hash.

Steps to Detect and Neutralize Divergence

1. Enforce Hermetic Toolchains: Use flags to lock tool versions across environments. Examples include --host_force_python=PY3 or specifying a fixed JDK via --host_javabase.
2. Isolate the Environment: Set BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 to prevent the auto-detection of varying host C++ compilers.
3. Compare Action Dumps: Run the build with --verbose_failures to identify the action ID of the miss. On both the CI runner and local workstation, execute:

   bazel dump --action_cache <action-id>

   Compare the Command line, Input manifest, and Environment sections to find the specific differing string.
4. Normalize Environment Variables: For any differing variables identified in the dump, explicitly define them in your .bazelrc using build --action_env=VAR=value to ensure consistency.

Impact of --remote_download_outputs=toplevel

The --remote_download_outputs=toplevel flag instructs Bazel to download only the final outputs of the requested targets. While intermediate actions may still experience cache misses and be executed locally, these misses are often obscured in the high-level build summary. Because the intermediate outputs are not downloaded to the local machine, the visibility of the "miss" is reduced, making it harder to diagnose nondeterminism without inspecting detailed execution logs.

Required Diagnostic Detail

If hashes remain inconsistent after normalizing toolchains and environment variables, please provide the output of bazel dump --action_cache <action-id> from both environments to determine if the divergence is rooted in the input manifest or the command line.