NixOS remote deployment failure due to binary cache mismatch

Short answer

Build the full system closure first, then push it to the target's store (or to a cache the target trusts) before running switch. The most reliable pattern is nixos-rebuild switch --target-host with --build-host (or plain nix copy --to ssh://target of the built toplevel), so the remote never has to resolve substitutes itself. dry-activate still realises the closure on the target — it only skips the activation step — so it is a valid connectivity probe but not a substitute for copying the paths.

Why local evaluation passes but remote switch fails

nix-instantiate only proves the expression evaluates to a derivation. Realisation is a separate step: every path in the closure must either be built or fetched from a substituter the target trusts. Likely explanations for your symptom:

• The target's substituters list in nix.conf lacks a cache you used locally (e.g. a project or internal cache), so paths fall back to source builds that then fail or are disallowed.
• The cache is listed but its public key is missing from trusted-public-keys, so valid paths are rejected on signature grounds.
• You built locally and assumed the paths exist on cache.nixos.org; anything built from an overlay or local source exists only in your local store.

These are the common causes, not confirmed facts for your case — the exact failure line from the remote build log would pin it down.

Recommended pre-flight procedure

1. Build the closure without switching:

   nixos-rebuild build --flake .#prod

2. Copy the realised toplevel to the target:

   nix copy --to ssh://prod-host ./result

   This transfers exactly the paths the target is missing, signed by your local key (add it to the target's trusted-public-keys or use a signed shared cache).
3. Then activate remotely:

   nixos-rebuild switch --flake .#prod --target-host prod-host

   With the closure already present, no cache fetching is needed.

Alternatively, skip manual copying entirely: nixos-rebuild switch --target-host prod-host --build-host localhost (or --use-substitutes off) builds locally and copies the closure over SSH in one step. Tools like deploy-rs or colmena implement the same build-then-push model.

Using dry-activate as a probe

On the target:

nixos-rebuild dry-activate --flake .#prod

This evaluates and realises the configuration — downloading substitutes — but stops before running the activation script, so the live system profile is untouched. A failure here during the fetch phase confirms a cache/signature problem before you risk a real switch. Note it does build or download paths into the store, so it is not fully side-effect free, but it never changes the running system.

To inspect cache health directly on the target:

nix-store --verify --check-contents /run/current-system
nix path-info -r ./result --store ssh://prod-host | head

The second command lists which paths of your new closure the target already has; anything missing must come from a substituter or nix copy.

Caveats

Exact flag names and defaults vary with Nix version (this assumes Nix 2.x with flakes enabled). If the remote error mentions "untrusted" or "signature" rather than 404s, fix trusted-public-keys rather than the substituter list. Verify against your deployment's actual nix.conf on both machines before changing production settings.