Diagnosing Failed NixOS Configuration Switches

The Problem: Failed System Activation

When running nixos-rebuild switch, you may encounter a scenario where the build succeeds (the Nix store is populated), but the activation fails. This leaves the system in a "half-switched" state: the system profile points to the new generation, but services may be crashing, kernel modules may not have loaded, or the network may be down.

The primary takeaway is that a successful build does not guarantee a successful activation. Because NixOS uses a declarative model, the transition from state A to state B relies on activation scripts that execute imperative commands (like restarting systemd units). If these scripts fail, your system state is inconsistent.

Diagnostic Matrix: Common Failure Symptoms

Symptom	Likely Cause	Diagnostic Focus
Build succeeds, but command hangs at "Switching to..."	Service timeout or deadlocked systemd unit	systemctl list-jobs
"Activation failed" error after build	Syntax error in a custom activation script or permission issue	/var/log/syslog or journalctl
System boots, but new config isn't active	Bootloader failed to update the default generation	/boot partition space or mount status
Disk full error during switch	Too many old generations (GC not run)	df -h /nix

Step-by-Step Recovery and Investigation

1. Identify the Activation Failure

If the nixos-rebuild command fails during the switch phase, check the system logs immediately. Run this command as a user with sudo privileges on the local terminal:

sudo journalctl -u nixos-rebuild -n 100

Look for errors related to systemd units failing to start or mount errors. If the logs are generic, check the specific service that failed to restart during the switch.

2. Verify Profile Consistency

Check if the system profile actually moved to the new generation. The system profile is a symlink in the Nix store.

ls -l /nix/var/nix/profiles/system

If the link points to the new generation but the system is behaving as if it is on the old one, the activation script failed before it could trigger the necessary service restarts.

3. Test for Resource Exhaustion

NixOS stores every generation in the /nix/store. If your root partition is full, the symlink update may fail or the bootloader may fail to write the new entry.

df -h /nix

If disk usage is above 90%, you must collect garbage to free space. Warning: This removes old generations and prevents rolling back to them.

sudo nix-collect-garbage -d

Fixes Based on Findings

Scenario A: The "Zombie" Service

If a service is refusing to stop, preventing the switch from completing, manually kill the process or force the unit to stop:

sudo systemctl stop [service-name]

Then, re-run sudo nixos-rebuild switch.

Scenario B: Configuration Regression

If the switch completed but the system is unstable (e.g., no WiFi, broken X11), use the atomic rollback feature. You can do this via the command line if the shell is still accessible:

sudo nixos-rebuild switch --rollback

If the system is unbootable, restart the machine and select the previous generation from the GRUB or systemd-boot menu. This changes the boot-time symlink back to a known-good state.

Limitations and Verification

The nixos-rebuild switch command cannot fix imperative changes. If you manually edited a file in /etc/ (outside of the Nix store), that change will persist until the NixOS activation script explicitly overwrites it. This can lead to "ghost configurations" where the system behaves differently than the .nix file suggests.

To verify a successful fix:

1. Run nix-env --list-generations to ensure the current generation number matches your latest build.
2. Check systemctl is-system-running to ensure no units are in a failed state.
3. Verify that the configuration file is correctly linked: ls -l /etc/nixos/configuration.nix.

Escalation Criteria

If the following conditions are met, the issue is likely beyond a simple configuration error and requires filesystem or bootloader repair:

• The boot menu is missing or does not list multiple generations.
• /nix/store reports read-only filesystem errors.
• The system fails to boot into any previous generation, indicating a corruption of the Nix store itself.