Diagnosing and Rolling Back System Changes with openSUSE Snapper and Btrfs

Recognizable Condition

After a zypper update, zypper dup, or a manual configuration edit, the system may fail to boot, critical services (such as sshd or NetworkManager) are missing, or files under /etc, /var, or /usr have disappeared. The root filesystem is formatted with Btrfs and Snapper is configured for the root subvolume (the default on openSUSE Tumbleweed and Leap).

Cause and Diagnostic Table

Ordered Checks

1. List available snapshots
   

   snapper list

   Run in a root shell (or via sudo). Confirm that snapshot IDs exist and are marked pre (before change) and post (after change) for the recent transaction. If the list is empty, Snapper is not configured for the root filesystem – this guide does not apply.

2. Inspect the latest snapshot
   

   snapper status <snapshot-id>

   Replace <snapshot-id> with the numeric ID of the most recent post snapshot. The output shows added, modified, or deleted files relative to the previous snapshot. Focus on /etc, /var, /usr, and /boot.

3. Compare snapshots to pinpoint changes
   

   snapper diff <pre-id> <post-id>

   Use the pre and post IDs surrounding the suspect transaction (e.g., a zypper dup). The diff displays unified diff hunks for text files and notes binary changes. Redirect to a file for easier review: snapper diff 123 124 > /root/diff-123-124.txt.

4. Verify Btrfs subvolume mount
   

   mount | grep btrfs

   Ensure the root subvolume (typically @ or @/.snapshots) is mounted at /. If the output shows /dev/nvme0n1p2 on / type btrfs (rw,relatime,subvol=@), the subvolume is correct. A missing or wrong subvol indicates a bootloader or fstab issue.

Fixes Tied to Findings

Unwanted System-Wide Changes (Rollback)

If snapper diff reveals numerous undesirable modifications across system directories, perform a full rollback to the pre snapshot:

snapper rollback <pre-snapshot-id>

What happens: Snapper creates a new read‑write snapshot based on the chosen pre snapshot, sets it as the new default subvolume, and updates the bootloader entry. The current (broken) state becomes a read‑only snapshot preserved for reference.

Required permissions: root. Risk: All changes made after the pre snapshot – including user data written to system directories – are lost. Back up /home or any data stored outside the snapshotted subvolumes before proceeding.

Verification: Reboot. The GRUB menu will show a new entry labeled openSUSE (snapshot <id>). Select it (or let the default boot). After login, run systemctl status foo.service and check that missing files are restored.

Single Package at Fault (Force Reinstall)

If the diff isolates the problem to one package (e.g., systemd or glibc), reinstall that package without a full rollback:

zypper install --force <package-name>

Run as root. This rewrites the package files from the repository while keeping the current snapshot. After reinstall, verify with rpm -V <package-name> (no output means all files match).

Individual File Deleted (Restore from Snapshot)

When only a few files are missing (e.g., /etc/ssh/sshd_config), copy them from the snapshot directory:

rsync -a /.snapshots/<snapshot-id>/snapshot/etc/ssh/sshd_config /etc/ssh/

Use rsync -a to preserve ownership and permissions. The snapshot path is /.snapshots/<id>/snapshot/ (the mounted read‑only snapshot). Adjust the source path for the exact file or directory.

Disk Space Exhaustion (Clean Old Snapshots)

If btrfs filesystem usage / shows Free near zero, delete the oldest snapshots to reclaim space:

snapper delete <old-snapshot-id>

Delete in pairs (pre + post) to keep the timeline consistent. Do not delete the most recent pre snapshot if you may need to rollback. After deletion, run btrfs filesystem usage / again to confirm freed space.

Escalation Criteria

• Rollback fails to boot – the GRUB entry points to a corrupted subvolume or the kernel/initrd mismatch persists. Proceed to boot from openSUSE installation media, choose Rescue System, mount the Btrfs root, and manually restore a known‑good snapshot or reinstall the kernel package (zypper install --force kernel-default).
• Snapshot metadata corrupted – snapper list returns errors or shows missing XML files under /.snapshots. Snapper cannot operate; use btrfs subvolume list / to locate readable snapshots and mount them manually for file recovery.
• Disk space cannot be freed – even after deleting snapshots, btrfs filesystem usage / shows no free space. The filesystem may be fragmented or have unreferenced extents. Run btrfs balance start -dusage=10 / (as root) to reclaim space; if that fails, a full backup and reformat may be necessary.

Limitations and Cautions

• Snapper snapshots only exist for Btrfs root filesystems. On ext4, XFS, or LVM without Btrfs, this recovery method is unavailable.
• Manually removing directories under /.snapshots deletes recovery points and may leave the system without a rollback option.
• Snapshot creation fails silently when the Btrfs volume is near capacity; always monitor free space with btrfs filesystem usage /.
• Rolling back reverts all changes after the target snapshot, including logs, databases, and user data stored in snapshotted subvolumes. Exclude /var/lib, /srv, or /home from snapshots via Snapper configuration if they contain critical mutable data.

Verification Checklist After Recovery

1. Run snapper list – confirm the new default snapshot is marked current.
2. Execute snapper status <current-id> – ensure no unexpected modifications remain in /etc, /var, /usr.
3. Reboot and verify the system reaches the graphical or multi‑user target.
4. Check the previously failing service: systemctl status <service-name>.
5. Confirm reinstalled package versions: zypper se -i <package-name>.