Question
Auth service metadata inconsistency during dynamic label hot-reloads
Rin Dey
0 reputation · 10 Sept 2022, 02:48 UTC
66.2K views
Loading…
0 reputation · 10 Sept 2022, 02:48 UTC
Teleport utilizes a declarative configuration model via auth.yaml to maintain repeatable development environments. While the auth service supports hot-reloads to apply state changes without full restarts, there is a behavior gap regarding how dynamic labels are processed during this transition.
In high-churn development environments where nodes are frequently registered and de-registered, the metadata state can become inconsistent if labels are updated in the configuration while the service is active. This creates a discrepancy between the teleport nodes list output and the desired state defined in the configuration file.
Does the Teleport auth service guarantee atomic updates for node labels during a SIGHUP triggered hot-reload? Is there a mechanism to ensure the metadata API synchronizes with the declarative state without requiring a full service restart?
32275 reputation · 10 Sept 2022, 11:56 UTC
Teleport’s SIGHUP reload updates only the declarative configuration in auth.yaml—cluster, auth, and proxy settings. Node labels are stored in the Teleport backend (etcd or the database) and are not part of the reloadable configuration. Consequently, a hot‑reload does not atomically change node labels, and the metadata API can temporarily show a state that differs from the file.
teleportauth.yaml with SIGHUP or tctl reload applies cluster‑level changes atomically.auth.yaml are ignored for label updates; the backend remains authoritative.tctl node update --labels are written directly to the backend and appear immediately in teleport nodes list.auth.yaml take effect, the auth service must be fully restarted (or a rolling restart in HA).When you need to change a node’s label, run:
tctl node update --labels key=value --node-name <node-name>
This writes the label directly to the backend and the change is visible in teleport nodes list immediately.
auth.yaml (e.g., adding a new static node with labels):
# Example for a 3‑node HA auth cluster
# This command restarts one node at a time
# ensuring the auth service remains available.
tctl cluster rolling-restart
To fine‑tune the recommendation, could you confirm whether your Teleport deployment is a single‑node auth service or a high‑availability cluster? The restart strategy differs between the two scenarios.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.