Managing Unattended Edge Nodes with openSUSE MicroOS Transactional Updates

Updating a fleet of remote edge devices or container hosts often presents a binary risk: the update succeeds, or the system enters a boot loop or loses network connectivity, requiring a physical site visit. Traditional package managers mutate the live filesystem, meaning a failed update can leave the OS in an inconsistent state.

openSUSE MicroOS solves this by treating the operating system as an immutable image. Instead of updating packages in place, it uses transactional updates—a process where changes are staged in a new Btrfs snapshot. The system only switches to this new version upon reboot. If the new version fails to boot or fails a health check, the system automatically rolls back to the last known good state.

How Transactional Updates Work

MicroOS employs a read-only root filesystem. While /etc and /var remain writable to preserve configuration and data, the core OS resides in a read-only layer. When you trigger an update, the system creates a new Btrfs snapshot, applies the changes there, and updates the GRUB bootloader to point to this new snapshot as the default for the next boot.

This architecture eliminates "package drift," where different nodes in a cluster end up with slightly different versions of libraries due to interrupted updates or manual interventions. Every node is guaranteed to be on a specific, verified snapshot.

Automating Safety with Health Checkers

The most powerful feature for unattended deployments is the health-checker.service. A successful boot doesn't always mean a successful update; for example, a kernel update might break a specific network driver or a container runtime.

MicroOS allows you to define custom scripts in /etc/health-checker.d/. After a reboot into a new snapshot, the system executes these scripts. If any script returns a non-zero exit code within the configured timeout window, the system assumes the update failed and automatically reboots back into the previous working snapshot.

Example: Implementing a Container Health Check

Suppose you are running a critical API gateway via Podman. You want the OS to roll back if the container fails to start after an update.

1. Create the check script
Run this as root on your MicroOS node:

cat << 'EOF' > /etc/health-checker.d/check-api-gateway.sh
#!/bin/bash
# Check if the api-gateway container is running
if podman ps --filter "name=api-gateway" --filter "status=running" | grep -q "api-gateway"; then
    exit 0
else
    echo "API Gateway failed to start! Triggering rollback."
    exit 1
fi
EOF

2. Set permissions
Ensure the script is executable:

chmod +x /etc/health-checker.d/check-api-gateway.sh

3. Verify and Test
To test the mechanism, you can manually trigger a transactional update and then intentionally stop the container before rebooting:

# Stage an update
transactional-update dup

# Stop the container to simulate failure
podman stop api-gateway

# Reboot the system
reboot

After rebooting, the health-checker.service will run, fail the script, and trigger a reboot back to the previous snapshot. You can verify this by checking the logs: journalctl -u health-checker.

Trade-offs and Limitations

The immutable nature of MicroOS introduces specific constraints that engineers must plan for:

• Reboot Requirement: Kernel updates and core OS changes always require a reboot to activate. There is no integrated live-patching for the kernel in MicroOS; maintenance windows must be scheduled.
• Read-Only /usr: Many legacy installers or monitoring agents attempt to write to /usr. These will fail. Such tools must be containerized or redirected to /var/usrlocal via bind mounts.
• Driver Complexity: Third-party kernel modules (like NVIDIA or ZFS) cannot be managed via standard DKMS because the root is read-only. You must either use the NVIDIA container toolkit or build custom images using the Kiwi image builder.

Practical Verification

To confirm your system is operating in transactional mode and to see your current snapshot history, run the following command on your terminal:

transactional-update status

This will list the current active snapshot and any staged updates waiting for a reboot. If you need to manually revert to a previous state without waiting for a health-check failure, you can select the previous snapshot directly from the GRUB boot menu during startup.