Migrating from k3OS: Decision Guide for Kubernetes OS Layer Replacement in 2024

The Decision: k3OS Is Deprecated — Plan Node Replacement Now

Rancher/SUSE archived the k3OS repository in 2022. The last release, v0.20.4-k3s1r0 (December 2021), ships Linux kernel 5.10.x and OpenSSL 1.1.1 — both end-of-life. No official security updates, SBOMs, or SLSA provenance exist. If you run k3OS today, you operate an unsupported supply chain. The only supported path is reinstalling nodes with a maintained minimal distribution and running k3s or RKE2 via systemd.

Constraints That Drive the Migration

• OpenSSL 1.1.1 EOL (September 2023): TLS 1.3 cipher suites and FIPS compliance are unavailable without rebuilding the entire rootfs.
• Kernel 5.10.x: No upstream LTS updates; live-patching (KernelCare, Canonical Livepatch) is a temporary bridge, not a solution.
• Cloud provider images unmaintained: AWS AMI, Azure VHD, and GCP images lack support for new regions or instance types.
• SELinux/AppArmor profiles outdated: Expect permission denials on container runtime transitions with k3s v1.25+.
• k3s version pin required: Existing clusters must stay on k3s v1.24.x (last tested) until nodes are replaced.

Target: replace all k3OS nodes within 6 months.

Supported Alternatives Compared

Distribution	Upgrade Mechanism	Config Format	iPXE Artifacts	Persistence Model	Best Fit
Flatcar Container Linux	rpm-ostree (atomic, rollback via rpm-ostree rollback)	Butane (.bu) → Ignition (.ign)	Signed flatcar_production_pxe.vmlinuz + initrd from stable.release.flatcar-linux.net	/usr/share/oem (config) + /var/lib/kubelet (data)	Teams wanting CoreOS-like immutability with direct cloud-init migration
Talos Linux	Talos API (image-based, talosctl upgrade)	Talos machine config (YAML, applied via talosctl apply-config)	Factory-built iPXE images via factory.talos.dev	EPHEMERAL partition (config) + /var/lib/kubelet (data)	Kubernetes-native OS; full API-driven lifecycle, no SSH
Ubuntu 22.04 LTS Minimal	apt + snap (traditional package updates)	cloud-init (native YAML)	Canonical netboot artifacts (ubuntu.com/download/cloud)	Standard /var/lib/rancher/k3s or /var/lib/rancher/rke2	Familiar ops, long LTS (2027), broad hardware support

Trade-offs

Flatcar

Closest mental model to k3OS: read-only /usr, config in /usr/share/oem, data in /var. Butane config syntax is a superset of Ignition; k3OS cloud-config translates with minor changes (see example below). rpm-ostree upgrades are atomic but require reboot. No SSH by default — access via coreos-installer or container tooling.

Talos

Purpose-built for Kubernetes; no general-purpose package manager, no SSH. Configuration is declarative machine config applied via talosctl. Upgrades are image swaps coordinated by the Talos API. Steeper learning curve but eliminates OS drift. Ideal if you want the cluster to manage its own nodes.

Ubuntu Minimal

Lowest friction for teams used to Debian/Ubuntu. cloud-init works unchanged. apt updates are familiar but not atomic — a failed kernel update can leave a node unbootable without console access. Largest attack surface of the three; requires ongoing patch management.

Concrete Migration: k3OS cloud-config → Flatcar Butane

k3OS used a k3os: top-level key in cloud-config. Flatcar uses Butane (YAML) which compiles to Ignition JSON. The mapping is straightforward:

Original k3OS cloud-config (example)

#cloud-config
k3os:
  hostname: worker-01
  install:
    device: /dev/sda
    silent: true
  k3s_args:
    - --node-label=role=worker
  ssh_authorized_keys:
    - ssh-ed25519 AAAAC3... user@laptop
  ntp_servers:
    - 0.pool.ntp.org
  dns_nameservers:
    - 1.1.1.1
  env:
    K3S_CLUSTER_SECRET: "${CLUSTER_SECRET}"
write_files:
  - path: /etc/rancher/k3s/registries.yaml
    content: |
      mirrors:
        docker.io:
          endpoint:
            - https://registry.internal

Equivalent Flatcar Butane config (worker.bu)

variant: flatcar
version: 1.0.0
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-ed25519 AAAAC3... user@laptop
storage:
  files:
    - path: /etc/hostname
      mode: 0644
      content:
        inline: worker-01
    - path: /etc/rancher/k3s/registries.yaml
      mode: 0644
      content:
        inline: |
          mirrors:
            docker.io:
              endpoint:
                - https://registry.internal
    - path: /etc/systemd/system/k3s.service.d/10-args.conf
      mode: 0644
      content:
        inline: |
          [Service]
          Environment="K3S_CLUSTER_SECRET=${CLUSTER_SECRET}"
          ExecStart=
          ExecStart=/usr/local/bin/k3s agent --node-label=role=worker
systemd:
  units:
    - name: systemd-timesyncd.service
      enabled: true
      dropins:
        - name: 10-ntp.conf
          contents: |
            [Time]
            NTP=0.pool.ntp.org
    - name: systemd-resolved.service
      enabled: true
      dropins:
        - name: 10-dns.conf
          contents: |
            [Resolve]
            DNS=1.1.1.1
    - name: k3s.service
      enabled: true

Compile and validate

# On a build machine with butane installed
butane --strict --files-dir=. worker.bu > worker.ign
# Check for syntax errors
ignition-validate worker.ign

Boot a test VM with the generated worker.ign via QEMU or your hypervisor's Ignition injection mechanism. Verify k3s starts and joins the cluster.

Network Boot Validation: iPXE for Flatcar

k3OS's k3os.install.* and k3os.mode kernel parameters map to Ignition config URL and Flatcar's flatcar.first_boot=1. Example iPXE script:

#!ipxe
set base-url https://stable.release.flatcar-linux.net/amd64-usr/current
kernel ${base-url}/flatcar_production_pxe.vmlinuz \
  initrd=flatcar_production_pxe_image.cpio.gz \
  ignition.config.url=https://config.internal/worker.ign \
  flatcar.first_boot=1 \
  console=tty0 console=ttyS0,115200n8
initrd ${base-url}/flatcar_production_pxe_image.cpio.gz
boot

Test with:

curl -f https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_pxe.vmlinuz
curl -f https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_pxe_image.cpio.gz

Both must return 200 OK. The Ignition URL must be reachable from the booting node (HTTP/HTTPS, no auth).

If You Must Keep k3OS Nodes Temporarily

1. Pin k3s to v1.24.17+k3s1 (last patch release for 1.24). Do not upgrade.
2. Enable kernel live-patching: apt install kernelcare (if kernel supports it) or Canonical Livepatch on Ubuntu-based kernels.
3. Run trivy image rancher/k3os:v0.20.4-k3s1r0 or grype rancher/k3os:v0.20.4-k3s1r0 weekly to track unpatched CVEs.
4. Monitor openssl version output — any service requiring TLS 1.3 or post-2023 cipher suites will fail.
5. Schedule node drain and replacement via your cluster's rolling update process; do not attempt in-place OS upgrade.

Verification Checklist Before Cutover

• k3os version on a test node confirms kernel 5.10.x and OpenSSL 1.1.1
• Flatcar/Talos iPXE artifacts fetch successfully (curl -f)
• Butane config compiles without warnings (butane --strict)
• Ignition config passes ignition-validate
• Test VM boots, k3s starts, node registers to cluster, workloads schedule
• CVE scan of replacement OS image shows zero critical/high vulnerabilities

Limitations

• This guide covers x86_64 only; ARM64 images exist for Flatcar and Talos but require separate validation.
• RKE2 on Flatcar/Talos uses the same systemd unit patterns; adjust ExecStart paths accordingly.
• Cloud-init migration for Ubuntu Minimal is nearly 1:1 but test runcmd and bootcmd sections — they execute in different stages.
• No rollback procedure for the OS replacement itself; node replacement is the rollback.