Diagnosing Bazel Non-Hermetic Sandbox Failures That Appear Only in CI

Builds pass on your laptop and then fail intermittently on CI with missing inputs, permission denied, or different output hashes. The useful takeaway is that the failure is almost always non-hermetic action inputs or environment leakage that is hidden locally by --spawn_strategy=standalone or a warm host filesystem.

Recognizable condition

Flaky CI errors that are stable locally. Typical signals after enabling stricter sandbox or moving to remote builders:

• Action fails with missing input file or file not found inside sandbox
• Test failures only under sandbox, or output hash mismatches on rebuild
• Permission denied for reads outside the declared input root
• Non-deterministic outputs between runs with identical sources

These appear most often with genrules, custom Starlark rules, and tools that assume host paths.

Cause diagnostic table

Ordered checks

1. Reproduce cleanly

Run in the repository root with developer permissions to execute Bazel.

bazel clean --expunge
bazel build //... --sandbox_debug

Risk: --sandbox_debug creates large copies under the output base. Ensure disk space.

Expected check: build reproduces the failure without a warm cache.

2. Capture sandbox contents

With --sandbox_debug Bazel leaves a sandbox root. Inspect the input root listing for the failing action.

ls -R <sandbox_root>/input

Compare the files present against the inputs declared by the rule. Missing files indicate undeclared inputs.

3. Inspect action environment

Compare local and CI environments.

bazel info environment

Run the same target on CI and locally and diff PATH, HOME and other variables the action may read. Action logs captured with --subcommands and --verbose_failures show the environment passed to the action.

4. Check runfiles and data

For tests, verify runfiles are used via $(location) and $(execpath) rather than absolute paths. Confirm data dependencies are declared and not accessed via host paths.

Fixes tied to findings

Absolute paths in genrules

Replace host absolute paths with Bazel substitutions and declare inputs explicitly.

genrule(
    name = 'process',
    srcs = [':input.txt'],
    outs = ['out.txt'],
    cmd = 'cat $(location :input.txt) > $@',
)

Do not embed /home/user/... or $(pwd). Add any tool binary to tools and declare it as an input.

Environment leakage

Wrap tools to avoid reading environment. Pass required values via action arguments, not via env.

bazel build //... --action_env=MY_FLAG=value

Remove reliance on HOME, USER, or PATH inside actions. If a tool needs PATH, provide a hermetic toolchain container.

Missing data/runfiles

Add missing files to srcs or data and use runfiles APIs in Starlark rules. For binaries, ensure runfiles are collected and accessed via runfiles library, not via host relative paths.

Toolchain hermeticity

Use --toolchains and define hermetic toolchain containers. Avoid repository rules that read host files.

Verification

• Run the failing target with --sandbox_debug and confirm the sandbox directory listing contains all declared inputs and no host paths leak in.
• Compare bazel info environment output between local and CI and diff action environment captured in logs to identify undeclared variable reads.
• Re-run the build after a fix with a clean workspace and verify repeated builds produce identical output hashes and pass under both standalone and sandbox strategies.

Escalation criteria

If failures persist after hermetic fixes with identical action input sets but different outputs, collect a minimal reproduction with the sandbox tarball, Bazel version, flags, and action graph summary. Do not disable sandboxing permanently; it hides non-hermetic behavior that will surface in remote execution.

Limitations

Sandbox flags and trace output format change across Bazel releases. Verify flag names for your version before relying on them. Remote execution and remote cache can mask hermeticity issues locally; test with local sandbox before assuming remote is correct.