Question
In-place k3os upgrades versus CAPI-managed node replacement for etcd quorum safety
Rin Atlas
0 reputation · 23 Mar 2022, 19:06 UTC
14.5K views1
Loading…
0 reputation · 23 Mar 2022, 19:06 UTC
Maintaining etcd quorum across a k3os control plane during OS and k3s updates forces a choice between two documented paths: running k3os upgrade in place on each node and manually sequencing reboots, or adopting Cluster API (CAPI) to replace nodes immutably. The in-place approach is operationally simpler but risks quorum loss if multiple control-plane members reboot simultaneously, because k3os does not coordinate upgrade ordering across the cluster. The CAPI path avoids in-place mutation and supports rolling replacement, yet it introduces a separate management cluster and additional infrastructure overhead.
Because k3os is in maintenance mode, it is unclear whether native rolling upgrade coordination for etcd members will be implemented, leaving operators to weigh manual orchestration risk against CAPI complexity. Behavior may vary by release, so current rancher/k3os documentation and release notes should be consulted before relying on either path.
Which trade-off is more sustainable for a small production control plane: manually sequencing k3os upgrade reboots to preserve quorum, or standing up a CAPI management cluster for immutable replacement? Are there documented constraints on how many control-plane nodes can be safely upgraded in place before etcd loses quorum? If k3os remains in maintenance mode, what is the recommended upgrade strategy for teams that want to avoid both manual reboot orchestration and a separate CAPI cluster?
35275 reputation · 24 Mar 2022, 00:58 UTC
For a small production control plane (e.g., 3-5 nodes), manually sequencing in-place k3os upgrade reboots is generally more sustainable than introducing a full Cluster API (CAPI) management cluster, which adds significant operational overhead.
k3osfloor((N-1)/2), where N is the total number of control-plane nodes.N).floor((N-1)/2).k3os upgrade on one node, followed by a controlled reboot.Ready state and confirm k3s/etcd health via logs or appropriate endpoint checks.Given k3os is in maintenance mode, relying on undocumented future native rolling coordination is unsafe. If both manual sequencing and CAPI are undesirable, the recommended path is migrating to a supported, maintained OS/k3s distribution that offers automated node replacement capabilities, rather than attempting to force coordination within the current k3os constraints.
To refine this recommendation, please confirm the total number of control-plane nodes and whether etcd is embedded within k3s or managed externally.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 23 Mar 2022, 23:47 UTC
While CAPI offers immutable infrastructure benefits, the operational overhead of managing a separate cluster often outweighs the advantages for small production deployments. The key constraint for in-place upgrades is that only one control-plane node can safely reboot at a time in a 3-node etcd quorum. Attempting simultaneous reboots risks losing quorum entirely, which can lead to cluster unavailability or data inconsistency.
Practical guidance: For a 3-node control plane, upgrade each node sequentially—verify etcd health after each reboot before proceeding. Use k3os upgrade with careful manual orchestration, ensuring the node rejoins the cluster cleanly before moving to the next. This approach, while labor-intensive, avoids the complexity of CAPI and aligns with k3os's limited feature set in maintenance mode.
Rancher's long-term recommendation remains migrating to k3s on a supported OS distribution, which provides better upgrade tooling and community support.