openSUSE MicroOS Transactional Updates: Architecture and Operational Reality

The Problem: Container Hosts That Update Themselves

Kubernetes worker nodes and edge devices need OS updates without scheduled maintenance windows or on-call engineers. Traditional package managers leave systems in intermediate states; a failed zypper up or power loss mid-transaction can brick the host. MicroOS solves this by treating the root filesystem as an immutable artifact: updates create a new Btrfs snapshot, the bootloader adds an entry, and a health-checker validates the boot before making it default. If validation fails, the previous snapshot boots automatically.

Requirements Driving the Design

• Zero-downtime atomic updates — the running system is never modified in place.
• Automatic rollback on failure — no manual GRUB selection or rescue mode.
• Minimal admin surface — no package layering, no manual edits that persist across reboots.
• Container-first workloads — host OS provides only kernel, container runtime, and kubelet; everything else runs in containers.

Target environments: edge gateways, IoT fleets, and Kubernetes clusters where SSH access is unreliable or nonexistent.

Smallest Suitable Design

Btrfs Subvolume Layout

The root filesystem is a read-only Btrfs subvolume named @. Mutable state lives in separate subvolumes:

Subvolume	Mount Point	Purpose
@	/	Immutable OS binaries, libraries, systemd units
@/var	/var	Logs, container images (/var/lib/containers), runtime state
@/etc	/etc	Machine‑specific configuration (managed by Ignition/Combustion or Salt)
@/home	/home	User data
@/var/lib/containers	/var/lib/containers	Container storage (Podman/cri‑o), isolated from OS snapshots

Run btrfs subvolume list / as root to confirm this layout on any MicroOS host.

Update Mechanism: transactional-update

transactional-update is a wrapper around zypper and snapper. It:

1. Creates a new read-write snapshot of @
2. Applies package updates inside that snapshot via zypper
3. Registers a new GRUB/BootLoader entry pointing to the snapshot
4. Leaves the current snapshot untouched

The update is not active until reboot. Daily checks are scheduled via transactional-update-du.timer (systemd timer).

Health Validation: health-checker

On every boot of a new snapshot, the health-checker systemd service runs. It executes scripts in /etc/health-checker.d/ (and built-in checks for network, kubelet, CRI). Only if all pass does it mark the snapshot as default via snapper undochange logic. Failure triggers automatic rollback: the next reboot loads the previous working snapshot.

Trust and Data Boundaries

Root is immutable. No rpm --install, no zypper in on the running system — changes vanish on reboot. Configuration lives in /etc (separate subvolume), managed declaratively:

• First boot: Ignition (cloud) or Combustion (ISO/USB) writes /etc
• Ongoing: Salt, Ansible, or transactional-update shell for interactive debugging (changes lost on rollback)

Container data is outside the transaction. /var/lib/containers is its own subvolume, excluded from snapshots. A rollback does not revert container images or volumes.

No trust in new binaries until health-checker passes. The new snapshot boots read-only; if kubelet fails to start or a custom check fails, the snapshot is never promoted.

Operational Checks and Diagnostics

Daily Workflow

# Preview what would change (run as root)
transactional-update --dry-run

# Apply updates and schedule reboot
transactional-update up
reboot

Post-Reboot Verification

# List recent snapshots; current boot marked with '*'
snapper list | head -10

# Health-checker status
systemctl status health-checker
journalctl -u health-checker -b

If systemctl status health-checker shows active (exited) with exit code 0, the snapshot is promoted. Non-zero exit means rollback occurred — check snapper list for the new default.

Snapshot Retention Policy

Configured in /etc/snapper/configs/root. Key parameters:

• NUMBER_LIMIT=10 — keep last 10 numbered snapshots
• TIMELINE_LIMIT_DAILY=7 — keep daily timeline snapshots for 7 days
• EMPTY_PRE_POST_CLEANUP=yes — remove empty pre/post pairs

Adjust for storage capacity; Btrfs space exhaustion is a real failure mode (see below).

Failure Modes and Mitigations

Failure Mode	Symptom	Mitigation
Health-checker false positive	Broken snapshot promoted; kubelet crashes post-boot	Write comprehensive checks: crictl pull test image, kubelet --version, custom app health endpoint
Btrfs space exhaustion	No space left on device during update; snapshot creation fails	Monitor btrfs filesystem usage /; tune NUMBER_LIMIT/TIMELINE_LIMIT; add snapper-cleanup.timer
Kernel regression	New kernel panics or breaks hardware; health-checker passes but workload fails	Rollback works but requires reboot. Pin kernel via zypper al kernel-default if needed
/etc drift lost on rollback	Manual vi /etc/kubernetes/kubelet.conf disappears after failed update	Enforce configuration management (Salt/Ansible); treat /etc as code

Concrete Example: Simulating a Failed Update

To verify rollback behavior before production deployment:

1. Create a failing health check:

   cat > /etc/health-checker.d/99-fail-test <<'EOF'
   #!/bin/sh
   exit 1
   EOF
   chmod +x /etc/health-checker.d/99-fail-test
   

2. Trigger update and reboot:

   transactional-update up
   reboot
   

3. Observe: system boots new snapshot, health-checker fails, GRUB menu shows previous snapshot as default on next boot.
4. Verify rollback:

   snapper list
   # Previous snapshot should now be marked as default
   

5. Clean up test:

   rm /etc/health-checker.d/99-fail-test
   transactional-update up
   reboot
   

Risk: This test causes a reboot cycle. Run only on non-production nodes or during maintenance window.

Conditions That Would Change the Design

• Need persistent package layering (e.g., install vim, tcpdump on every host) → use openSUSE Leap/Tumbleweed with traditional zypper or rpm-ostree.
• Full OS customization at build time (custom kernel, pre-baked containers) → build a custom MicroOS image with kiwi.
• Third-party kernel modules required (NVIDIA, ZFS, proprietary drivers) → MicroOS kernel updates break them. Options: Tumbleweed kernel (newer, same issue), custom kernel package, or switch to Leap.
• FIPS 140-2 compliance → MicroOS does not ship FIPS-validated modules. Use openSUSE Leap with FIPS-enabled packages.
• Zero-reboot requirement → Transactional updates require reboot. kexec is not enabled by default; consider traditional OS with live patching if reboot is unacceptable.

Hard Constraints to Accept

• Btrfs is mandatory for root. No ext4, XFS, or ZFS support for transactional snapshots. Verify storage hardware compatibility (no hardware RAID without Btrfs awareness).
• Combustion/Ignition runs once. First-boot configuration only. Subsequent changes need configuration management or accept loss on rollback.
• Health-checker runs only at boot. Runtime failures after promotion (OOM kill, certificate expiry) require external orchestration (Kubernetes node auto-repair, watchdog) — MicroOS does not re-evaluate.

Verification Checklist for New Deployments

1. Boot MicroOS VM or test hardware.
2. Run transactional-update --dry-run; confirm package list looks correct.
3. Run transactional-update up && reboot.
4. Post-reboot: snapper list | head -5 shows new snapshot as default.
5. systemctl status health-checker shows success.
6. btrfs subvolume list / confirms expected subvolume layout.
7. cat /etc/snapper/configs/root retention matches operational policy.

If all seven pass, the transactional update pipeline is functional. Document the health-checker scripts in your runbook — they are the contract between OS and workload.