Fedora Silverblue rpm-ostree Architecture: Design, Trust Boundaries, and Operational Checks

Requirements Driving the Design

Fedora Silverblue targets a specific problem: deliver a base operating system that cannot be mutated by routine operations, while still letting administrators install additional RPMs and run containerized workloads. The requirements are:

• OS immutability — the shipped /usr tree must remain bit-for-bit identical to the signed OSTree commit.
• Safe rollback — a failed upgrade must be reversible by selecting the previous boot entry.
• Atomic, offline-capable updates — the new deployment stages completely before any reboot; no live patching of kernel or userspace.
• RPM ecosystem compatibility — layered packages use the same RPM database and dependency solver as traditional Fedora.
• Container-first application model — Flatpak handles user-facing apps; podman/Toolbox handle development environments.

These constraints rule out traditional dnf-based in-place upgrades and lead directly to the OSTree + rpm-ostree composition model.

Smallest Suitable Design

The deployed system consists of three pieces:

1. Read-only base image — an OSTree commit mounted at /usr (read-only) and /boot (read-only except for bootloader config). The commit is identified by a content-addressed checksum and signed by the Fedora release key.
2. Writable system state — /etc and /var live on a separate read-write filesystem (typically the root LV or a dedicated partition). /etc receives a three-way merge on each upgrade: pristine upstream /etc from the new commit, the previous commit's /etc, and local modifications. /var (including /home) is entirely outside the image — data persists across upgrades and rollbacks unchanged.
3. User-layered packages — rpm-ostree install <pkg> composes a new local commit that adds the requested RPMs on top of the base commit. This composition runs with the invoking user's privileges but writes to /ostree/deploy/fedora/deploy/<checksum>.<serial>. The result appears as a new bootloader entry.

Flatpak applications install into /var/lib/flatpak (system) or ~/.local/share/flatpak (user) and never touch the base image.

Trust and Data Boundaries

Boundary	Trust Root	Data Flow	Mutation Authority
Base OS image	Fedora GPG-signed OSTree commits in fedora/40/x86_64/silverblue ref	Pulled from mirror → local OSTree repo → deployed read-only	Only the Fedora release engineering pipeline
Local composition	User's RPM database + Fedora package signatures	User runs rpm-ostree install → new commit staged in /ostree/deploy	Any user with wheel/sudo (via polkit)
/etc configuration	Three-way merge algorithm (ostree)	Upstream /etc from new commit + previous commit + local edits → merged result	Automatic on upgrade; manual on conflict
/var and /home	None (user data)	Persists untouched across all operations	Applications and administrators

The critical insight: local composition runs with user privileges but writes into the system deployment directory. This means a compromised user account with sudo can stage a malicious commit. The bootloader menu (GRUB with BLS entries) is the only user-visible selection point — an attacker who can write a crafted commit and a BLS entry could persist across reboots. Mitigation: restrict sudo, verify ostree log --verify before reboot, and consider UEFI Secure Boot with enrolled MOK for kernel command line locking.

Operational Verification Checklist

Run these on the booted Silverblue system (Fedora 40+). All commands require a regular user; rpm-ostree operations invoke polkit for root actions.

1. Confirm booted deployment matches staged state

rpm-ostree status
ostree admin status

Both should show the same commit checksum under Booted deployment. If they differ, a staged upgrade awaits reboot.

2. Dry-run an upgrade

rpm-ostree upgrade --check

Exit code 0 = updates available; 1 = up to date; 2 = error. No changes are written. Review the package list before committing.

3. Stage the upgrade

rpm-ostree upgrade

This downloads the new commit, performs the /etc three-way merge, writes a new deployment directory, and adds a BLS entry. Disk space risk: /var (for OSTree repo cache) and /boot (for kernel/initramfs) must have room. A full /boot blocks staging — clean old kernels with rpm-ostree cleanup -p first (requires reboot to free space).

4. Verify commit signatures before reboot

ostree log --verify fedora/40/x86_64/silverblue

Every commit should show Good signature from Fedora. An unsigned or malformed commit is rejected by rpm-ostree automatically, but manual verification catches supply-chain issues.

5. Inspect /etc merge result

ostree admin config-diff

Shows the three-way merge output versus pristine upstream. Run this after staging but before reboot to catch silent config drops. If conflicts occurred, the merge aborts and rpm-ostree upgrade exits non-zero — resolve manually in /etc then re-run.

6. Validate systemd units post-merge

systemd-analyze verify

Catches unit file syntax errors introduced by the /etc merge or layered packages.

7. Test rollback

Reboot, interrupt GRUB, select the previous deployment entry. Confirm:

• System boots to the prior commit
• rpm-ostree status shows the older checksum as booted
• Local /etc changes persist (they live in /etc, not the image)
• /var data is intact (databases, logs, user files)

Failure Modes and Mitigations

Failure Mode	Symptom	Recovery	Prevention
/etc merge conflict	rpm-ostree upgrade exits with conflict markers in /etc	Edit conflicted files, remove markers, re-run upgrade	Audit ostree admin config-diff after each upgrade; minimize local /etc edits
Layered package dependency conflict	Composition fails with RPM dependency error	Remove conflicting layer (rpm-ostree uninstall) or adjust package set	Prefer Flatpak/Toolbox; layer only kernel modules, drivers, CLI tools
Disk full on /var or /boot	Staging fails mid-download or initramfs generation	rpm-ostree cleanup -p (removes old deployments), then reboot	Monitor df -h /var /boot; allocate 2GB+ for /boot
Unsigned/malformed OSTree commit	ostree pull or rpm-ostree upgrade rejects commit	Verify mirror integrity; check ostree remote gpg-import	Use default Fedora mirrors; don't add untrusted remotes
Kernel/driver regression	Hardware fails after reboot into new deployment	Rollback via GRUB; file kernel bug	Test in Toolbox/VM first; maintain at least two boot entries
SELinux relabel delay on rollback	Boot pauses 10–30s at "Relabeling filesystem"	Wait; it's automatic	Expect delay; don't hard-reset. Local policy modules require relabel every boot after rollback.

Concrete Example: Layering a Kernel Module

Scenario: You need the akmod-nvidia package for proprietary GPU support. This is a legitimate layering case — the kernel module must match the booted kernel exactly.

# Stage the layer (requires reboot)
rpm-ostree install akmod-nvidia

# Verify it appears in staged deployment
rpm-ostree status
# Look under Layered packages in the staged (not booted) entry

# Reboot into new deployment
systemctl reboot

# Confirm module loads
modinfo nvidia
lsmod | grep nvidia

Risk: Each kernel update from Fedora produces a new base commit. The akmod rebuild triggers on boot via akmods.service. If the rebuild fails, X/Wayland won't start. Rollback restores the previous kernel + working module. Limitation: Layered packages increase the attack surface and break the immutability guarantee for the layered portion. Prefer toolbox or a container for userspace tools; reserve layering for kernel-space necessities.

Conditions That Would Change This Design

The current architecture is stable but not final. These upstream shifts would require redesign:

• Unified Kernel Images (UKI) — replacing BLS entries with a single signed EFI executable bundling kernel, initramfs, and kernel command line. This moves trust verification earlier (UEFI Secure Boot validates the UKI) and eliminates GRUB/BLS as an attack surface. Fedora is experimenting with UKI in rawhide; Silverblue would adopt it once tooling matures.
• systemd-sysext for user-space extensions — replacing rpm-ostree layering with portable, verified extension images mounted via systemd-sysext. This preserves base immutability while allowing additive user-space packages without local composition. Would require Flatpak-style signatures for extensions.
• Mandatory FDE with TPM2-bound keys — encrypting /var (and possibly /etc) with keys sealed to TPM2 PCRs. This changes the data boundary: rollback would require PCR state matching the deployment, complicating recovery. Current design assumes /var is unencrypted or encrypted with a static passphrase.
• OSTree content-addressable storage (CAS) deduplication — changing the repository layout from hardlink-based to CAS. Would reduce disk usage for multiple deployments but requires migration tooling and breaks assumptions about /ostree/repo structure.
• Flatpak portal permissions replacing SELinux for app confinement — shifting the confinement model from MAC labels to capability-based portals. Would reduce SELinux policy complexity in the base image but requires portal support in all Flatpak runtimes.

Limitations and Practical Verification

• Rollback restores only the OS image. /var state (database schemas, log rotation state, application caches) moves forward. Applications must handle schema downgrades gracefully — test this in staging.
• Bootloader entries accumulate. rpm-ostree cleanup -p removes unpinned deployments but requires a reboot to free /boot space. Pin the current and previous deployment: rpm-ostree pin 0 (current) and rpm-ostree pin 1 (rollback target).
• No live patching. Kernel and userspace updates require reboot. Plan maintenance windows; rpm-ostree upgrade + reboot takes 2–5 minutes on NVMe, longer on HDD.
• Verification is manual by default. Automate the checklist via a systemd timer or Ansible playbook: run rpm-ostree upgrade --check, alert on exit 0, then stage and verify signatures before scheduled reboot.

Verification Procedure for New Deployments

1. Boot fresh Silverblue 40+ ISO; complete first-boot setup.
2. Run rpm-ostree status — confirm booted deployment matches ostree admin status.
3. Run rpm-ostree upgrade --check then rpm-ostree upgrade — verify new deployment appears staged.
4. Run ostree log --verify fedora/40/x86_64/silverblue — confirm GPG-verified commits.
5. Run ostree admin config-diff — inspect /etc merge.
6. Run systemd-analyze verify — validate units.
7. Reboot; select previous deployment in GRUB — confirm rollback works and local /etc changes persist.
8. Test layering: rpm-ostree install htop — verify it appears under Layered packages and persists across a subsequent rpm-ostree upgrade + reboot.

If any step fails, do not promote the deployment to production. The architecture's safety depends on completing this loop before relying on the system.