NixOS Bootloader Rollback: Recovering from a Failed System Rebuild

You edit /etc/nixos/configuration.nix, run nixos-rebuild switch, and the next boot hangs or a critical service fails. The useful takeaway: NixOS does not overwrite your running system in place. Every successful rebuild creates a separate, read-only system generation, and the bootloader menu lists those generations as boot entries. You can recover by selecting an earlier generation at boot, often without external rescue media.

Desired Outcome

Restore a bootable NixOS system after a configuration change causes a failure, using the bootloader's generation entries rather than reinstalling or manually repairing files.

Prerequisites

• Root or sudo access to edit /etc/nixos/configuration.nix and run rebuild commands.
• A bootloader managed by NixOS, typically GRUB or systemd-boot, with a menu accessible at boot.
• At least one prior system generation present on disk.
• Awareness that files changed directly outside the Nix store (for example, editing /etc/ssh/sshd_config by hand) are not tracked and may be discarded by the next rebuild.

How NixOS Generations Map to the Bootloader

NixOS evaluates configuration.nix into store paths under /nix/store. When nixos-rebuild switch succeeds, it updates the system profile symlink and, depending on your bootloader settings, writes a new menu entry. GRUB entries often appear as NixOS - Configuration <number>; systemd-boot creates similarly versioned entries. Because the old generation remains on disk until garbage collection, the bootloader can chain-load it directly.

Procedure: Apply a Change and Prepare for Rollback

1. Edit the configuration. Make a targeted change in /etc/nixos/configuration.nix. For example, to add a kernel parameter:

   {
     boot.kernelParams = [ "quiet" ];
   }

   Keep the change small enough that you can reason about what might fail.

2. Validate before switching (optional but safer). Run a dry build to catch syntax errors without touching the running system:

   sudo nixos-rebuild dry-build

   This evaluates and builds but does not activate the result. If it fails, the current system is untouched.

3. Activate the configuration. Run as root (or with sudo):

   sudo nixos-rebuild switch

   Expected check: the command should complete without error and print the new generation number. If the build phase fails, the running system remains on the previous generation.

4. Confirm the generation exists. List system generations to ensure a new entry was created:

   nix-env --list-generations

   Note: this lists generations for the current user profile by default. On some setups you may need to target the system profile explicitly, for example nix-env --list-generations --profile /nix/var/nix/profiles/system. Verify the exact profile path on your installation if the output looks unexpected.

Expected Checks After Reboot

• Bootloader menu: Reboot and hold the appropriate key (often Esc, F12, or Shift for GRUB) to open the menu. You should see multiple NixOS entries corresponding to generations.
• System health: After boot, run systemctl is-system-running. A healthy system returns running or degraded. If it returns starting indefinitely or units are failed, the new generation may be faulty.
• Service status: Check the specific service you changed, for example systemctl status sshd.

Recovery Options

Option 1: Boot a Previous Generation

If the system boots to a broken state or kernel panic, reboot into the bootloader menu and select an earlier generation. This does not change any files on disk; it only changes which generation the kernel loads. Once booted, you can inspect logs and fix configuration.nix before switching again.

Option 2: Roll Back the Current Session

If the system is running but unstable, and you want to revert to the previous generation immediately:

sudo nixos-rebuild rollback

This switches the active profile to the generation that was current before your last switch. It changes system state, so run it only when you have decided to abandon the current generation. After it completes, reboot to ensure the bootloader default points to the restored generation.

Option 3: Rescue a Broken Bootloader

If you edited boot.loader.grub or boot.loader.systemd-boot settings and the menu no longer appears, the bootloader configuration itself may be invalid. In that case, boot from a NixOS installation medium, mount your root filesystem, and chroot into the installed system to run nixos-rebuild boot or repair the configuration. This is a higher-risk operation; validate bootloader changes with nixos-rebuild dry-build before switching whenever possible.

Option 4: Reclaim Disk Space

Generations accumulate in /nix/store. After you have confirmed a stable configuration, remove old generations to free space:

sudo nix-collect-garbage --delete-older-than 7d

Replace 7d with the retention window you prefer. This permanently removes rollback targets, so do not run it until you are confident the current generation is healthy.

Limitations and Risks

• Atomicity boundary: Atomicity protects the running system during the build phase. It does not guarantee that a successfully built generation will boot correctly.
• Imperative drift: Manual edits to files under /etc that are not managed by NixOS will be overwritten or ignored by declarative updates.
• Bootloader fragility: Incorrect bootloader settings can produce an unbootable system. Treat bootloader changes as higher-risk than service changes.
• Single-step rollback: nixos-rebuild rollback moves back only one generation. To return to an older generation, select it from the boot menu or use the bootloader entry directly.

Practical Verification

Before relying on this workflow, perform a test on a non-production machine: create a generation, reboot, confirm the menu shows at least two entries, then run sudo nixos-rebuild rollback and reboot again. If the older generation boots and systemctl is-system-running reports a healthy state, your rollback path is working. Because NixOS options evolve, verify command flags and profile paths against the documentation for your specific release before performing critical repairs.