Changing the default boot‑failure rollback policy for NixOS generations

Answer

Yes – NixOS should attempt an automatic rollback to the previous generation after a boot failure, but only when the failure is detected reliably and a small safety limit is respected.

Confirmed facts

• Each NixOS generation gets its own GRUB menu entry; manual rollback is done via nixos-rebuild switch --rollback which toggles the GRUB saved_entry variable.
• The system.maxGenerations option limits how many generations are kept; older generations are garbage‑collected and cannot be booted.
• Mutable directories such as /etc and /var are not versioned, so a rollback only rewinds the immutable Nix store.
• GRUB already provides a mechanism to boot a specific entry by setting saved_entry; this can be scripted from an early‑boot service.

Likely explanation / recommended policy

1. Detect a boot failure: if the system does not reach multi-user.target within a configurable timeout (e.g., 90 s) or if the emergency shell is activated, set a flag /run/boot-failed.
2. On the next boot, an early‑boot service checks for that flag. If present, it looks at the generation metadata (e.g., a timestamp stored in /nix/var/nix/profiles/per-user/root/channels or a simple file /run/boot-success.gen) to find the most recent generation that booted successfully before the current one.
3. If such a generation exists and has not been garbage‑collected (its path still under /nix/var/nix/profiles), write its GRUB entry number to saved_entry and clear the flag.
4. Otherwise, do nothing and let the current generation boot (preserving the ability to debug).

Interaction with system.maxGenerations

The rollback service must skip generations that have been removed by the garbage collector. It can do this by checking whether the generation's profile directory still exists; if not, it continues searching older generations until either a valid one is found or the limit imposed by system.maxGenerations is reached.

Safeguards against infinite rollback loops

• Store a retry counter in /run/boot-failed-count. Increment it each time an automatic rollback is attempted.
• If the counter reaches a configurable threshold (default 2), abort further automatic rollbacks and require manual intervention.
• Clear the counter and the failure flag after a successful boot (detected by reaching multi-user.target).
• Optionally, require that the target generation has a recorded success timestamp older than the current failure, ensuring we do not bounce between two generations that both failed.

Implementation steps (for a NixOS module)

# /etc/nixos/configuration.nix
{ config, lib, pkgs, ... }:
{
  boot.loader.grub.enable = true;
  # Enable the automatic rollback service
  services.boot-failed-rollback.enable = true;
  # Optional tuning
  services.boot-failed-rollback.timeout = 90;   # seconds to wait for multi-user.target
  services.boot-failed-rollback.maxRetries = 2; # give up after N attempts
}

The corresponding module would:

• Create a oneshot service that runs late in the boot process to set /run/boot-failed if multi-user.target is not reached within the timeout (using systemd timeout logic or a simple watchdog script).
• Create another early‑boot service (ordered before local-fs.target) that reads the flag, consults /run/boot-failed-count, selects a valid previous generation, writes to GRUB's saved_entry via grub-reboot, and clears the flag on success.
• Provide the two tunable parameters as module options.

Missing diagnostic detail

To finalize the recommendation we need to know: what maximum number of consecutive boot failures should trigger giving up on automatic rollback? If the environment prefers a more aggressive policy (e.g., only one failure) or a more conservative one (e.g., three failures), the suggested default of 2 can be adjusted.