Choosing the Right NixOS Generation Rollback Strategy for Production

Why Rollback Matters in NixOS

Every nixos-rebuild that switches or tests a configuration writes a new, immutable closure under /nix/var/nix/profiles/system--link. The bootloader (systemd‑boot, GRUB, or U‑Boot) creates a matching entry so you can pick any generation at boot. This design gives instant, safe rollbacks, but it also consumes disk space and requires a clean garbage‑collection policy. In a production environment you need a predictable, automated approach that guarantees you can recover from a bad update while keeping the system lean.

The Decision: How to Retain or Prune Generations

When you ask "how should I manage generations?" you are really choosing between three axes:

1. How many generations to keep (time‑based vs count‑based)
2. Which tool to trigger deletion (automatic GC vs manual command)
3. How to expose the current generation to the bootloader (systemd‑boot vs GRUB)

Below is a compact table that contrasts the supported options and their trade‑offs for a typical server or workstation.

Options Comparison

Option	Retention Policy	Tool	Bootloader Interaction	Pros	Cons
Automatic GC with nix.gc.automatic = true	Time‑based (e.g., nix.gc.dates = "90d") or count‑based via systemd-boot.configurationLimit	Systemd‑boot or GRUB manages entries; GC removes unreferenced generations	Hands‑off, consistent with Nix store; protects current booted system	Requires sufficient /boot space; may keep too many generations if not tuned
Manual GC with nix-collect-garbage	Explicit --delete-older-than or --dry-run checks	Bootloader entries remain until you delete them manually	Full control over when deletions happen; safe for auditing	Risk of forgetting to run GC; manual entry cleanup needed
Flake‑based generations (e.g., nixos-rebuild --flake .#host)	Same as above but generations share the same profile; flake.lock pins inputs	Bootloader sees the same entries; rollback works across flake revisions	Deterministic builds; easier CI integration	Requires understanding of flake semantics; lock updates need coordination
Filesystem snapshots (ZFS/Btrfs) + Nix generations	Snapshots provide an extra safety net; Nix generations still exist	Bootloader unaffected; snapshots are independent of boot entries	Double protection; can recover even if Nix store is corrupted	Complexity of snapshot management; extra disk usage

Key Constraints to Keep in Mind

• /boot size: Each generation duplicates the kernel and initrd. On /boot < 512 MiB, you must limit entries (e.g., systemd-boot.configurationLimit = 10) or move them to a larger partition.
• Current booted generation: The symlink /run/booted-system protects the active closure from GC. Do not delete it manually.
• Flake lock updates: When using flakes, the lock file pins the exact inputs. Rolling back to a previous generation automatically restores the pinned state.
• Remote builders: nixos-rebuild --build-host still creates local generations on the builder; the built closure is copied over before switching.

Concrete Implementation: A Production‑Ready Rollback Workflow

Below is a step‑by‑step guide you can adopt on a server that uses systemd‑boot. Adapt the paths if you use GRUB or U‑Boot.

1. Configure Automatic GC and Generation Limits

# /etc/nixos/configuration.nix
{ config, pkgs, ... }: {
  nix.gc.automatic = true;
  nix.gc.dates = "90d";          # keep generations for 90 days
  boot.loader.systemd-boot.configurationLimit = 12; # max 12 entries
  boot.loader.systemd-boot.canTouchEfiVariables = true;
}

After editing, rebuild and switch:

sudo nixos-rebuild switch

Verify the boot entries:

ls -l /boot/loader/entries/

You should see system-.conf files for each generation.

2. Deploy a New Configuration Safely

1. Build only (no switch):

   sudo nixos-rebuild build
   

2. Inspect the new closure:

   diff -r /run/current-system /nix/var/nix/profiles/system--link
   

3. If satisfied, switch:

   sudo nixos-rebuild switch
   

Immediately after the switch, confirm the system is healthy:

systemctl list-units --failed

3. Test the Rollback Path

Reboot into the previous generation to ensure rollback works. If you prefer a quick test without reboot, use the command‑line switch:

sudo nixos-rebuild switch --rollback

After reboot, check that the expected services are running:

systemctl status nginx

To confirm the generation number, run:

nixos-rebuild list-generations

The output should show the current generation marked with <current> and the previous one as <previous>.

4. Verify Garbage Collection Protects the Booted Generation

Run a dry‑run to see what would be deleted:

nix-collect-garbage --dry-run

Look for the line containing /run/booted-system – it should not appear in the list of deletions. This confirms that the system you just booted is safe from GC.

5. Clean Up Old Generations Manually (Optional)

If you need to free space immediately, delete generations older than a threshold but keep the current one:

sudo nix-collect-garbage --delete-older-than 30d

Afterward, verify the boot entries again:

ls -l /boot/loader/entries/

Common Pitfalls and How to Avoid Them

• Running GC on a system with an active custom /boot partition: Ensure /boot has enough free space for at least configurationLimit kernel copies.
• Deleting the wrong symlink: Never touch /run/current-system or /run/booted-system; use nix-collect-garbage only.
• Forgetting canTouchEfiVariables on UEFI systems: Without this flag, systemd‑boot will not update the EFI NVRAM entries automatically.
• Mixing flake and non‑flake builds: Generations created by a flake share the same profile; rolling back will revert the entire configuration, including the flake lock.

Conclusion

By combining automatic GC with a count‑based bootloader limit, you get a hands‑off, safe rollback mechanism that keeps your /boot partition from filling up. The manual GC path gives you auditability when you need to scrub space quickly. Flakes add reproducibility, and filesystem snapshots provide an extra safety net. Pick the combination that matches your operational risk profile and storage constraints.