Declarative Machine Configuration with Talos: Changing Kernel Parameters via API

Problem: Changing Kernel Settings Across a Cluster

In a Kubernetes cluster built on Talos Linux, you often need to adjust low‑level kernel parameters (e.g., increasing shared memory limits) without logging into each node via SSH. Manual edits are error‑prone, break immutability guarantees, and complicate rollback when a change prevents the node from booting.

How Talos Applies Declarative Updates

Talos exposes a machine‑configuration API that accepts YAML or JSON payloads through the talosctl CLI or directly over HTTP. When you send a new configuration, Talos:

1. Writes the updated state to an inactive partition.
2. Performs an atomic switch to that partition on the next reboot.
3. Keeps the previous working partition available for instant rollback if the new config fails to boot.

Because the OS image is immutable and all system components run as containers, the change is consistent across nodes and can be managed with GitOps tools.

Worked Example: Raising kernel.shmmax

Suppose you want to increase the kernel shared‑memory limit from the default 32 MiB to 256 MiB on all worker nodes.

1. Prepare the configuration file

Create a file named worker-shmmax.yaml with the following content:

machine:
  kernel:
    sysctl:
      kernel.shmmax: 268435456  # 256 MiB in bytes

2. Apply the change

Run the command from a workstation that has network access to the Talos API endpoint (typically port 50000). You need talosctl installed and authenticated with a token that has the node:write privilege.

talosctl -n <endpoint>:50000 apply-config --file worker-shmmax.yaml

This sends the payload to the API; Talos validates the YAML, writes the new system state to the inactive partition, and schedules the switch for the next reboot.

3. Verify the pending configuration

Before rebooting, you can inspect what Talos will apply:

talosctl -n <endpoint>:50000 get config

The output should show the kernel.shmmax value under machine.kernel.sysctl. No reboot has occurred yet, so the running kernel still uses the old value.

4. Reboot and confirm the new value

Trigger a reboot (via talosctl reboot or your infrastructure’s power‑control). After the node comes back, check the running kernel:

talosctl -n <endpoint>:50000 get kernel

Or, if you have SSH access for debugging only, run inside the node:

sysctl kernel.shmmax

You should see kernel.shmmax = 268435456. If the node fails to boot, Talos automatically boots from the previous healthy partition, giving you a chance to fix the config.

5. Rollback (if needed)

To revert to the last known good state manually, run:

talosctl -n <endpoint>:50000 rollback

This tells Talos to switch back to the previously active partition on the next reboot. Because the rollback only changes the boot target, it is safe to execute even if the node is currently running.

Trade‑offs and Limitations

• Learning curve: You must become familiar with talosctl and the partition‑based upgrade model, which differs from traditional yum/apt workflows.
• Scope: Talos is optimized for Kubernetes host nodes; using it as a general‑purpose desktop or server OS may feel restrictive.
• Atomicity window: The new configuration only takes effect after a reboot. For parameters that can be changed at runtime (e.g., some sysctl values), you still need to reboot to see the change, though the atomic partition switch guarantees consistency.

Actionable Closing

Adopting Talos’ API‑driven declarative configuration lets you treat infrastructure as code while preserving the safety guarantees of an immutable OS. Start by version‑controlling a base machineconfig.yaml in your Git repository, automate updates with Flux or Argo CD, and validate each change in a lab cluster using the verification steps above. When a change fails, the built‑in rollback ensures the cluster remains operational, giving you confidence to evolve low‑level settings without the fear of bricking nodes.