Talos Linux gRPC API and External Config Management Interoperability

Question

Integration Boundary: Talos API and External Orchestrators
Talos Linux replaces traditional SSH access with a gRPC API for all management tasks. When integrating external configuration management tools for cluster bootstrapping, the system relies on a reconciliation loop to converge the node to a desired state defined in the machine configuration.
A challenge arises when mapping external state definitions to the two-tier hierarchy of Talos machine and cluster configurations. While talosctl apply-config --dry-run allows for syntax validation, it does not simulate the full reconciliation behavior or the potential for reboot loops if conflicting configurations are applied across the hierarchy.
Given the requirement for TLS certificates to access the API, there is uncertainty regarding the optimal method for external tools to handle secret rotation without losing management access.

How does the Talos API handle conflicting state definitions between the machine-level and cluster-level configurations during a reconciliation cycle?
What is the recommended mechanism for external orchestrators to rotate TLS certificates via gRPC without interrupting the management plane?

Tasadduq Burney · Answer

How Talos resolves machine vs. cluster configuration conflicts
Confirmed behavior: during each reconciliation loop Talos first loads the cluster‑level configuration, then overlays the machine‑level configuration. Machine‑level fields take precedence over conflicting cluster fields.
Likely explanation: the merge is a simple overlay; if the resulting combined configuration would trigger a reboot loop or fail validation, Talos detects the invalid state and rolls back to the last known good configuration, preventing a persistent loop.
Verification steps for this case

Create a machine config that sets a field (e.g., kernel.modules) to a value different from the cluster config.
Run talosctl apply-config --dry-run to ensure syntax is valid.
Apply the config with talosctl apply-config.
Check the effective configuration via talosctl get config and confirm the machine‑level value appears.

If you observe a rollback, inspect journalctl -u talos-api for messages indicating the invalid state detection.
Rotating TLS certificates via the gRPC API without interruption
Confirmed behavior: external orchestrators can rotate TLS certificates by updating the trust section (machine‑level or cluster‑level) with the new certificate bundle and issuing talosctl apply-config. The API keeps the old listener active while loading the new bundle, providing dual‑cert support during the reload.
Likely explanation: the gRPC server loads the new certificate bundle into memory, keeps the existing listener alive, and once the new certificate is verified it gradually migrates connections, avoiding any drop in management access.
Steps for safe rotation

Generate a new leaf certificate signed by the existing CA (do not change the CA unless you also update the trust bundle).
Update the machine configuration’s trust.certificates field (or the cluster trust if you prefer) with the new bundle.
Apply the change: talosctl apply-config.
From a client, connect to the API endpoint and verify the presented certificate has changed; the connection should remain alive.
Optionally, check journalctl -u talos-api for log entries about cert loading and listener migration.

If you encounter authentication errors after the switch, verify that clients trust the CA used to sign the new leaf certificate.

Talos Linux gRPC API and External Config Management Interoperability

Integration Boundary: Talos API and External Orchestrators

1 answer

How Talos resolves machine vs. cluster configuration conflicts

Verification steps for this case

Rotating TLS certificates via the gRPC API without interruption

Steps for safe rotation

Your perspective belongs here.

0 question comments