Talos Linux: Declarative Machine Configuration with talosctl Apply-Config

Why Talos Uses a Declarative API Instead of SSH

Talos Linux was built for cloud‑native workloads. It removes the traditional shell, SSH daemon, and root password to harden the OS core. All administrative actions go through a gRPC API that listens on port 50000. The talosctl CLI talks to that API over mutual TLS, so the node never exposes a login prompt or any of the usual shell tools.

The result is a read‑only root filesystem and a single source of truth: a YAML machine configuration. When you change that file, Talos reconciles the node to match it, rebooting only when necessary. This model lets cluster operators apply the same configuration to thousands of nodes with a single command.

Getting Started – The talosctl apply-config Workflow

Below is a minimal, but complete, example that shows how to create a new worker node configuration and apply it. The steps assume you already have a Talos cluster running and the talosctl binary installed on your workstation.

1. Generate a machine‑config YAML template – Talos ships a helper to create a skeleton file.

   # Run on any node that already has Talos installed
   $ talosctl get machineconfig --output yaml > base-config.yaml
   

   This command fetches the current configuration from a node and writes it to base-config.yaml. You can edit this file to change desired settings.

2. Modify the role and hostname – For a worker node, change the nodeName and role.

   nodeName: worker-01
   role: worker
   kubernetes:
     kubelet:
       extraArgs:
         v: 2
   

   The kubernetes.kubelet.extraArgs section is optional; it shows how to increase verbosity for debugging.

3. Apply the new configuration to the target node – Replace <NODE_IP> with the IP of the node you want to configure.

   $ talosctl apply-config \
       --insecure \
       --nodes <NODE_IP> \
       --config base-config.yaml
   

   Flags explained:

   • --insecure – Skips certificate verification; use only in test environments.
   • --nodes – Target IP or hostname. The CLI uses the node’s public key from the cluster’s certificate authority.
   • --config – Path to your YAML file.

   After the command finishes, the node will reboot if the change required a kernel or init‑system update. Verify the applied config with:

   $ talosctl get machineconfig <NODE_IP>
   

How the API Works Under the Hood

Talos’s talosctl uses a gRPC client that connects to the node’s API endpoint. The connection is established over mutual TLS: the client presents a certificate signed by the cluster CA, and the node presents its own signed certificate. Once authenticated, the client sends a SetMachineConfig RPC with the YAML payload. The node validates the config, writes it to a read‑only overlay, and triggers the talosd service to apply changes.

Because the root filesystem is immutable, any runtime changes (e.g., installing a package) must go through talosctl or a container image rebuild. This design eliminates drift between nodes and ensures that a failed node can be replaced by re‑applying the same config.

Common Mistakes and How to Avoid Them

• Using the wrong certificate bundle – Talos expects the cluster CA and node key to be in the default ~/.talos directory. If you point talosctl to a different bundle, the API call will fail with “certificate unknown”. Verify with talosctl get nodes before applying.
• Modifying immutable fields – Some fields, like nodeName or role, cannot be changed after the node has joined the cluster. Attempting to change them will trigger a full node reset. Check the talosctl get machineconfig output for the immutable flag.
• Applying a broken YAML – A syntax error in the YAML will cause the API to reject the config. Talos validates the schema before writing. Run talosctl validate-config base-config.yaml to catch errors early.
• Ignoring reboot requirements – Some changes, such as updating the kernel version, force a reboot. If you apply a config that triggers a reboot, the node will disconnect temporarily. Plan changes during maintenance windows.
• Overriding the API port – The API listens on 50000 by default. If you change api.port in the config, you must also update the talosctl command’s --api-port flag.

Limitations of the Declarative Model

• No interactive shell – Traditional troubleshooting commands (e.g., top, ps, ls) are not available. Use talosctl logs and talosctl get machineconfig for diagnostics.
• Limited runtime package management – Installing packages on the host requires rebuilding a container image or using talosctl apply-config to reference a new image. There’s no apt install on the node.
• Dependency on mutual TLS – If the CA is compromised or a certificate expires, all nodes lose API access. Rotate certificates carefully and monitor expiry dates.
• Reboot on certain changes – Not all configuration changes are hot‑reloadable. Kernel upgrades, network interface changes, and certain security settings require a reboot.

Practical Verification Checklist

1. Confirm the node is reachable: talosctl get nodes.
2. Validate the YAML before applying: talosctl validate-config base-config.yaml.
3. Apply the config: talosctl apply-config --nodes <NODE_IP> --config base-config.yaml.
4. Verify the applied config: talosctl get machineconfig <NODE_IP> and compare against the source file.
5. Check logs for any errors: talosctl logs <NODE_IP>.

Following this checklist ensures that changes are applied cleanly and that the node remains in a known state.

When to Use Talos’s Declarative API Over Traditional SSH

• Large‑scale, immutable infrastructure where drift must be avoided.
• Security‑first environments that cannot expose a root shell.
• Cluster operators who prefer versioned configuration files.
• CI/CD pipelines that automate node rollout via YAML manifests.

In contrast, if you need ad‑hoc debugging or quick package installs on a single machine, a traditional SSH‑enabled distro may be more convenient. Talos is designed for environments where the entire node lifecycle is managed declaratively.

Next Steps

With the basic apply-config workflow in place, you can explore more advanced features:

• Using talosctl apply-config --dry-run to preview changes.
• Automating the process with Ansible or Terraform via the Talos provider.
• Integrating with GitOps tools like Flux to keep node configs in sync with a Git repo.

Each of these builds on the same declarative principle: the node’s state is always derived from a single, versioned YAML file.