Managing Package Layering with rpm-ostree in Fedora Silverblue and Kinoite

The Problem: Installing Software on a Read-Only Filesystem

Fedora Silverblue and Kinoite utilize an immutable root filesystem. Unlike traditional Fedora Workstation, where dnf modifies the system in real-time, the /usr directory in these variants is read-only. This prevents accidental system corruption and ensures atomic updates, but it means you cannot simply install a system-level utility or driver using standard package managers.

The solution is package layering via rpm-ostree. Layering allows you to request specific RPM packages be merged into the read-only base image. Instead of modifying the current running system, rpm-ostree creates a new deployment (a bootable system snapshot) that includes your requested packages. This change only takes effect after a reboot.

How Package Layering Works

When you layer a package, rpm-ostree does not "install" it in the traditional sense. It calculates a new system state by combining the base OS commit (the immutable image provided by Fedora) with your list of layered packages. This new state is staged as a separate deployment. Upon reboot, the bootloader points to this new deployment.

Layered packages persist across base OS updates. When you run a system upgrade, rpm-ostree pulls the latest base commit and automatically re-applies your layered package list on top of it.

Worked Example: Layering a System Utility

While most applications should be installed via Flatpak or run in a container (Toolbox/Distrobox), some tools—such as distrobox itself or specific VPN clients—require system-level integration.

Step 1: Install the package
Run the following command from your terminal. This requires sudo permissions:

sudo rpm-ostree install distrobox

Step 2: Verify the pending change
Check the status to see that a new deployment is staged but not yet active:

rpm-ostree status

Step 3: Apply the change
Reboot the system to boot into the deployment containing the new package:

systemctl reboot

Step 4: Confirm installation
After rebooting, verify the package is active in the current deployment:

rpm -q distrobox

The output of rpm-ostree status will now show distrobox listed under the "LayeredPackages" section of the booted deployment.

Operational Constraints and Limitations

Layering is a powerful tool, but it should be used sparingly to maintain the benefits of an immutable system.

Performance and Update Risks

• Boot Times: Every layered package slightly increases the complexity of the deployment merge.
• Upgrade Conflicts: If a layered package depends on a specific version of a library that is updated or removed in a new base OS commit, the upgrade may fail or require manual intervention.
• Disk Space: Each deployment retains a copy of the base image and layered packages. Use rpm-ostree cleanup -m to remove old, unused deployments, though it is recommended to keep at least one for rollback.

Filesystem Restrictions

Because /usr is read-only, any layered package that attempts to write data to /usr/local or /opt at runtime will fail. You must use /var (which is writable) or configure bind mounts for these paths.

Non-Interactive Transactions

Unlike dnf, rpm-ostree is declarative. You cannot use interactive flags like --allowerasing or --best to resolve dependencies on the fly. If a dependency conflict occurs, you must resolve it by adjusting your layer list or using an override.

Comparison: Layering vs. Containers

To decide whether to layer a package or use a container, refer to the following criteria:

Criteria	rpm-ostree Layering	Container (Toolbox/Podman)
Scope	System-wide / Kernel level	User-space / Application level
Persistence	Survives reboots and updates	Persistent within container volume
Activation	Requires reboot	Instant start
Use Case	Drivers, VPNs, System utilities	Compilers, IDEs, CLI tools

Rollback and Recovery

If a layered package causes system instability or prevents booting, you can revert to the previous known-good state.

Option A: GRUB Menu
During boot, select the previous deployment entry from the GRUB menu to boot into the system state as it existed before the last rpm-ostree install.

Option B: Command Line
If the system is bootable but unstable, run the following to set the previous deployment as the default for the next boot:

sudo rpm-ostree rollback && systemctl reboot

Verification Checklist

• Check Current State: Run rpm-ostree status to identify the booted commit and active layers.
• Preview Updates: Run rpm-ostree upgrade --preview to see if upcoming base updates will conflict with your layered packages.
• Test Read-Only Guard: Run touch /usr/testfile; it should return Read-only file system, confirming the immutable protection is active.