Atomic System Rollbacks in NixOS: A Practical Engineering Decision

The Problem: Configuration Drift and Unrecoverable Upgrades

When managing a fleet of Linux servers, administrators often apply patches, tweak kernel modules, or adjust service settings manually. Over time these ad‑hoc changes accumulate, leading to configuration drift. A failed upgrade can leave the system in a non‑bootable state, and reproducing the last known‑good configuration requires painstaking manual reconstruction.

Declarative System Definition in NixOS

NixOS treats the entire operating system as a pure function of a single configuration file, /etc/nixos/configuration.nix. Every aspect—kernel version, loaded modules, filesystem layout, user accounts, and daemon settings—is expressed as options within Nix modules. Changing an option does not mutate the running system; instead, the Nix evaluator derives a new system derivation that describes the desired state from scratch.

Atomic Generations and Instant Rollbacks

Each successful nixos-rebuild operation creates a new generation stored as a symlink under /nix/var/nix/profiles/system. The bootloader (GRUB or systemd‑boot) points to the current generation via the current symlink. Switching to a different generation merely updates that symlink, which is an atomic, side‑effect‑free operation. If the new generation fails to boot, the administrator can revert to the previous generation by restoring the symlink, effectively rolling back the whole system—including kernel and services—instantly.

Worked Example: Adding a Kernel Module and Rolling Back

1. Prepare the change. Edit /etc/nixos/configuration.nix and add the module to the kernel packages list, for example:

boot.kernelPackages = pkgs.linuxPackages_6_6; // example version
boot.extraModulePackages = [ pkgs.r8168 ];   // add a real‑world NIC driver

Save the file.

1. Test the build without switching. Run as root (or with sudo):

sudo nixos-rebuild test

This command evaluates the configuration, downloads any missing sources, and builds the new system derivation. If the build succeeds, you see a message like “building Nix...” and the derivation is stored in the Nix store. No symlink is changed, so the running system stays untouched.

Apply the new generation. If the test passes, switch to the new generation:

sudo nixos-rebuild switch --upgrade

The --upgrade flag also updates the Nixpkgs channel to the latest revision before building. The command creates a new generation symlink and updates the bootloader entry.

Detect a failure and roll back. Suppose the new kernel causes a boot failure. After the system fails to start, you can boot into the previous generation from the bootloader menu (often accessed by holding Shift or Esc and selecting the entry labeled “NixOS – ”). Alternatively, from a rescue shell you can run:

sudo nixos-rebuild switch --rollback

This command moves the current symlink back to the prior generation, making the bootloader point to the known‑good kernel. The operation is instantaneous; no files are deleted or mutated.

Verify the rollback. Once the system is back online, confirm the running kernel matches the previous generation:

uname -r

Compare the output with the kernel version stored in /run/current-system/kernel (a symlink that points into the Nix store of the active generation). If they match, the rollback succeeded.
Trade‑offs and Practical Limitations

Learning curve. The Nix language and its module system require study; a mis‑typed option can prevent a successful rebuild, leaving no new generation to switch to.
Disk usage. Each retained generation keeps its closure of packages in the Nix store. On a server with frequent upgrades, this can consume tens of gigabytes.
Garbage collection. To reclaim space, administrators should run periodic garbage collection, for example:

sudo nix-collect-garbage -d
Optionally, enable automatic collection with the Nix configuration options nix.gc.automatic = true and nix.gc.max = 5 to keep only the five most recent generations.
Actionable Closing
Start by adopting the atomic generation workflow on a test machine or virtual instance:

Make a small change to configuration.nix.
Run sudo nixos-rebuild test to verify the build.
If the test succeeds, apply it with sudo nixos-rebuild switch.
Keep the system current by occasionally running sudo nixos-rebuild switch --upgrade.
Schedule a weekly garbage‑collection job (e.g., via a systemd timer) to control disk growth.

By treating the whole OS as a declarative, version‑controlled artifact, you gain the ability to reproduce any known‑good state instantly and to roll back upgrades without manual intervention—turning what used to be a risky, error‑prone process into a reliable engineering practice.