Diagnosing k3OS A/B Atomic Upgrade Failures

k3OS uses a dual-partition (A/B) atomic upgrade strategy: the inactive partition receives the new image, the bootloader is updated, and a health check marks the partition "good" before the next reboot. When k3os upgrade reports success but the system boots the previous version—or k3os version still shows the old release—the rollback mechanism has rejected the new partition. This guide provides a structured diagnostic path to identify the root cause, apply the correct fix, and verify the upgrade is committed.

Recognizable Condition

• Upgrade command exits with success, yet the system reboots into the old OS partition.
• k3os version displays the legacy version despite a matching ISO checksum.
• The new partition exists but the k3s API is unreachable after reboot.
• Console shows a brief switch attempt followed by an automatic rollback.

Cause and Diagnostic Table

Potential Cause	Recognizable Indicator
Bootloader entry not updated	grub.cfg (or UEFI boot order) still points to the old partition label.
Post-install health check failure	k3os-service did not mark the new partition "good"; rollback triggered.
Missing k3os.upgrade.device in cloud-init	Installer wrote to the wrong disk (first disk alphabetically when multiple present).
Overlayfs mount failure on new partition	/var/lib/rancher/k3s corrupted or unreadable after upgrade.
Network timeout during ISO fetch	Partial write left the partition unbootable or incomplete.

Ordered Validation Checks

Run these checks in sequence on the booted (old) system to pinpoint the failure point.

1. Verify bootloader configuration

   # UEFI systems
   efibootmgr -v
   
   # GRUB environment
   grub2-editenv list

   Confirm the next-boot entry targets the new partition label (e.g., K3OS_B).

2. Inspect partition labels

   blkid | grep K3OS

   Both K3OS_STATE (active) and the inactive label (K3OS_STATE_2 or similar) must exist.

3. Examine the inactive partition rootfs

   mount /dev/disk/by-label/K3OS_STATE /mnt
   ls /mnt/k3os/system
   umount /mnt

   Look for a complete k3os/system directory with the expected kernel and initrd.

4. Review installer logs from the previous boot

   journalctl -u k3os-upgrade -b -1

   Search for health-check failures, ISO download errors, or bootloader update messages.

5. Verify container health (if you managed to boot the new partition)

   crictl ps -a

   Ensure k3s control-plane containers are running and not in crash-loop.

Targeted Fixes per Finding

1. Bootloader failed to switch

If the new partition files are intact but the bootloader wasn't updated:

# Force bootloader rewrite
k3os upgrade --force-bootloader

# Or manually set the success flag (GRUB)
grub2-set-bootflag boot_success=1

2. False-positive health-check rollback

Only bypass the health check after confirming the ISO SHA256 matches:

sha256sum k3os.iso  # verify
k3os upgrade --no-health-check --image file:///path/k3os.iso

3. Wrong target disk due to missing cloud-config device

Add the explicit device to your cloud-init user-data and re-trigger:

# cloud-config snippet
k3os:
  upgrade:
    device: /dev/sda
    strategy: dual-partition

4. Overlayfs corruption on the state partition

Wipe the state partition and re-run the upgrade (data loss: workload data on state partition will be erased):

wipefs -a /dev/disk/by-label/K3OS_STATE
k3os upgrade --image file:///path/k3os.iso

5. Network timeout during ISO fetch

Use a local ISO file to eliminate transfer issues:

k3os upgrade --image file:///path/k3os.iso

Escalation Criteria

• Both partitions unbootable – requires PXE/iPXE re-provisioning.
• Repeated health-check failures on clean media – indicates hardware disk or UEFI firmware issue.
• Upgrade loops more than three times – open a Rancher/k3OS GitHub issue with sosreport output.

Rollback Verification

After applying a fix, reboot and watch the console for:

k3os: booting from partition B
k3os: upgrade successful

Then run k3os version to confirm the new release. No firmware-setup reboot is required.

Automation Hint

Embed these settings in cloud-init to make upgrades self-verifying:

k3os:
  upgrade:
    strategy: dual-partition
    health_check: true
# Add a systemd oneshot that runs after boot
# /etc/systemd/system/k3os-upgrade-verify.service
[Unit]
Description=Verify k3OS upgrade
After=network-online.target
[Service]
Type=oneshot
ExecStart=/bin/sh -c 'k3os version | grep -q "$(cat /etc/k3os/version)" || exit 1'
[Install]
WantedBy=multi-user.target

Enable it with systemctl enable k3os-upgrade-verify.service.

Cautions and Limitations

• k3OS is archived (June 2022); no security patches after v0.20.7. Consider migrating to Talos, Flatcar, or Bottlerocket for production.
• Dual-partition layout requires GPT with an ESP; MBR or single-partition installs cannot roll back.
• Health check only validates k3s API responsiveness, not workload pods – a false "good" is possible.
• Cloud-init device parameter is ignored if multiple disks exist and wipe: true is not set; installer picks the first disk alphabetically.
• Overlayfs corruption on power loss during write – enable sync mount option via k3os.mount_options in config.
• Upgrading across more than two minor versions (e.g., 0.18 → 0.20) may skip required migration steps; step through each release.

Verification Steps

1. Manually boot each partition via GRUB menu and run k3os version to confirm both kernels/userlands match the expected release.
2. Simulate power loss during upgrade (pull power at "Syncing filesystem..." console message) and verify automatic rollback on next boot.
3. Check fwupdmgr get-devices for pending firmware updates that could interfere with ESP writes.
4. Run k3os upgrade --dry-run --image <url> and parse JSON output for target_partition and health_check_url fields.
5. Capture sosreport --batch --tmp-dir /var/tmp after a failed upgrade; attach to any upstream issue.