Atomic Upgrades and Rollbacks on Rocky Linux with rpm‑ostree

Why Atomic Upgrades Matter on Rocky Linux

In an immutable server environment, a single failed package update can render a machine unusable. rpm‑ostree solves this by treating the whole system as a single, bootable snapshot. When you upgrade, a new snapshot is created; if anything goes wrong, the previous snapshot is still bootable and can be restored with a single command. This guarantees that a bad update never leaves the system in a broken state.

Desired Outcome

Perform a transactional system upgrade on Rocky Linux 8 or 9, verify the new snapshot, and know how to roll back if necessary.

Prerequisites

• Running Rocky Linux 8.8+ or 9.3+ with rpm‑ostree installed (default in the minimal image).
• Root or sudo privileges to run upgrade commands.
• At least 2 GB free space on the /usr partition and 1 GB free on /boot for snapshot metadata.
• Optional: a custom rpm‑ostree repository configured in /etc/rpm-ostree.conf.

Procedure

1. Check Current Snapshot and Available Space

   Run:

   sudo rpm-ostree status
   

   This shows the active commit hash and any pending upgrades. Verify that /usr and /boot have sufficient free space:

   df -h /usr /boot
   

2. Prepare a Test Upgrade

   To see what will happen without affecting the current system, request the next available commit:

   sudo rpm-ostree upgrade --dry-run
   

   Review the list of packages that will be added or removed. If the output looks correct, proceed.

3. Execute the Upgrade

   Run the actual upgrade. Replace <repo> with your repository URL or omit it to use the default:

   sudo rpm-ostree upgrade --repo=<repo>
   

   During this step, rpm‑ostree creates a new bootable snapshot while keeping the old one. The command blocks until the transaction completes. On success, the system will reboot into the new snapshot automatically.

4. Verify the New Snapshot

   After reboot, confirm you are on the new commit:

   rpm-ostree status
   

   The output should show Active commit: <new-hash>. You can also inspect the GRUB menu:

   cat /etc/grub2.cfg | grep 'menuentry'
   

   Each entry corresponds to a snapshot. The one without a suffix is the current boot.

5. Rollback if Needed

   If the new snapshot fails to boot or you detect a critical issue, roll back with:

   sudo rpm-ostree rollback
   

   This command switches the bootloader to the previous snapshot and reboots automatically. To verify, run rpm-ostree status again; the Active commit should now be the older hash.

6. Clean Up Old Snapshots (Optional)

   Snapshots accumulate over time. To free space, delete snapshots older than a specific date or count:

   sudo rpm-ostree cleanup --keep=<count>
   

   Replace <count> with the number of recent snapshots you want to keep. This operation is safe but irreversible.

Expected Checks

• After upgrade, rpm-ostree status must report the new commit as Active and Pending as empty.
• Bootloader entries should list both the new and old snapshots.
• Running systemctl status for critical services (e.g., sshd) should show active (running).

Recovery Options

• Rollback – sudo rpm-ostree rollback restores the previous snapshot.
• Boot to a Specific Snapshot – During boot, press Esc or Shift to access the GRUB menu and select the desired entry.
• Reinstall the Base Image – If snapshots become corrupted, reinstall the Rocky Linux base image via ISO or network install.

Limitations and Caveats

• Disk Space – Insufficient space on /boot or /usr will abort snapshot creation. Monitor with df -h.
• Third‑Party DKMS Modules – Kernel module drivers built via DKMS are not automatically rebuilt after a kernel snapshot switch. Rebuild manually with dkms autoinstall after a rollback.
• Non‑Atomic Packages – Some packages that modify the filesystem outside of RPM may not be fully captured in a snapshot, leading to inconsistencies.

Practical Verification Checklist

1. Run rpm-ostree status – confirm current commit.
2. Check /etc/grub2.cfg – ensure both snapshots listed.
3. Verify critical services – e.g., systemctl status httpd.
4. Test rollback – sudo rpm-ostree rollback and verify the commit change.

Conclusion

Using rpm‑ostree on Rocky Linux provides a robust, atomic upgrade path that protects against failed updates. With a few commands you can create, validate, and roll back snapshots, keeping your immutable servers reliable and secure.