k3os rootfs overlay update and unintended k3s version drift

Question

Rootfs Overlay Update and k3s Version Alignment
The goal is to keep a specific k3s major version stable across OS upgrades. The update.strategy setting defaults to a best‑effort alignment that does not explicitly pin the k3s binary, so an OS refresh can pull in a newer k3s release without notice. This drift can leave a node running an incompatible k3s binary after a rootfs overlay commit, potentially breaking cluster operations.
Key uncertainty: the current k3os configuration does not expose a standard way to bind a particular k3s semver to the update cycle. When external registries are used, the update pipeline does not validate that the fetched k3s image matches the declared version.
Specific questions to clarify this behavior:

Is there a documented method to pin a k3s major version in /etc/k3os/config.toml that is respected during rootfs overlay updates?
Does the update.strategy setting support an explicit k3s semver constraint, and if so, how is it expressed?
When pulling k3s images from a custom registry, what mechanisms exist to automatically verify image compatibility with the OS version?

Muntasir · Answer

Managing k3s Version Stability in k3os

In k3os, there is currently no documented method to pin a specific k3s semver within /etc/k3os/config.toml that will override the binary bundled with the OS image during a rootfs overlay update. The update.strategy governs the OS image refresh process, not the internal versioning of the k3s binary itself.

Analysis of Version Drift
Because k3os is an immutable operating system, the k3s binary is treated as a core component of the OS image rather than a decoupled package. When an OS update occurs, the underlying image is replaced. While the rootfs overlay preserves your configuration and data, it does not protect the k3s binary from being updated to the version bundled with the new OS release.

Regarding custom registries: k3os typically trusts the image tag provided by the update orchestrator. There is no native, automatic mechanism within the k3os boot process to verify that a fetched image's internal k3s version matches a declared version constraint.

Recommended Mitigation Steps
Since native pinning is unavailable, use these steps to maintain version alignment:

Verify Bundled Versions: Before triggering an OS update, check the k3os release notes for the target OS version to identify exactly which k3s version is bundled.
Manual Version Validation: Run the following command before and after the update to detect drift:
k3s --version

Controlled Rollouts: Perform OS updates on a single node first to verify that the bundled k3s version is compatible with your current cluster state before updating the rest of the fleet.

Required Diagnostic Detail
To determine if a more specific workaround exists for your environment, please provide the exact version of k3os you are currently running. Certain legacy or experimental builds may have different configuration schemas.

k3os rootfs overlay update and unintended k3s version drift

Rootfs Overlay Update and k3s Version Alignment

1 answer

Managing k3s Version Stability in k3os

Analysis of Version Drift

Recommended Mitigation Steps

Required Diagnostic Detail

Your perspective belongs here.

0 question comments