Diagnosing and Fixing Talos Linux Node "NotReady" Status

Problem Overview

A Talos node that reports NotReady in Kubernetes is usually symptomatic of a bootstrap failure. The kubelet, Talos’s container runtime agent, cannot start because one or more prerequisites are missing or mis‑configured. Common culprits are network mis‑settings, disk partitioning errors, or an incorrect cluster endpoint in the machine config.

Common Causes

Cause	Typical Symptom	Root Cause
Network configuration mismatch	Node cannot reach API server; kubelet logs show connection timeouts.	Incorrect CIDR, gateway, or DNS in the machine config.
Disk partitioning failure	Boot or /var/lib/kubelet directories missing; kubelet fails to start.	Wrong device path or insufficient space during apply-config.
Missing or wrong cluster endpoint	Authentication errors in kubelet logs; node remains NotReady.	Cluster endpoint or TLS settings absent or incorrect.
Boot‑loop / systemd failures	Repeated talosctl logs entries of failed services.	Corrupted system services or mis‑applied firmware updates.

Diagnostic Flow

Follow the table below to decide which check to run based on the observed symptom.

Observed Symptom	First Check	Next Step if Issue Persists
Kubelet logs show connection timeouts or DNS errors	Verify network config via talosctl logs kubelet	Check CIDR, gateway, and DNS entries.
Missing /var/lib/kubelet or /var/log directories	Inspect disk partitions with talosctl logs and talosctl health	Re‑apply config with correct device paths.
Authentication failures in kubelet logs	Confirm cluster endpoint in machine config.	Re‑apply config with updated endpoint.
Repeated systemd failures in logs	Run talosctl health for component status.	Consider talosctl reset if corruption suspected.

Step‑by‑Step Checks

1. Confirm node connectivity

   # Run from any management machine
   talosctl health --node <node-ip>
   

   All components should return OK. If kubelet is FAILED, proceed to step 2.

2. Inspect kubelet logs

   # View last 200 lines
   talosctl logs kubelet --node <node-ip> --tail 200
   

   Look for errors like dial tcp or authentication failed. This indicates network or endpoint issues.

3. Validate network configuration

   # Show current network settings
   talosctl logs network --node <node-ip>
   

   Check that cidr, gateway, and dns match the cluster’s overlay network. If mismatched, prepare a corrected machine config:

   {
     "machine": {
       "network": {
         "interfaces": [
           {
             "name": "eth0",
             "addresses": ["10.42.0.5/24"],
             "routes": [{"cidr": "0.0.0.0/0", "gateway": "10.42.0.1"}]
           }
         ],
         "dns": {"servers": ["10.42.0.1"]}
       }
     }
   }
   

   Apply with talosctl apply-config.

   talosctl apply-config --node <node-ip> --config <config-path>
   

   After applying, re‑run talosctl health to verify the kubelet starts.

4. Check disk partitions

   # List partitions and mount points
   talosctl logs systemd --node <node-ip> | grep -i "mount"
   

   If /var/lib/kubelet is missing, the boot partition may have been mis‑created. Re‑apply config with correct device paths or run talosctl reset to wipe and start fresh.

5. Verify cluster endpoint

   # Inspect machine config for endpoint
   cat <config-path> | grep -i "endpoint"
   

   It should point to the API server’s IP/hostname and port. If absent, add:

   {
     "machine": {
       "clusterEndpoint": "https://api.cluster.local:6443"
     }
   }
   

   Re‑apply and check kubelet logs for authentication success.

Fixes Tied to Findings

• Network mismatch: Update machine.network section and re‑apply.
• Disk partition error: Use talosctl reset to clear corrupted state, then re‑apply with correct bootDevice and rootDevice paths.
• Endpoint mis‑config: Add or correct clusterEndpoint in machine config and apply.
• Boot‑loop: If systemd shows repeated failures, consider a full reset or replacing the firmware image.

Escalation Criteria

If after the above steps the node remains NotReady and logs show persistent authentication or connectivity errors, consider:

• Verifying the control plane’s API server is reachable from the node’s network (e.g., talosctl exec <node-ip> -- ping -c 4 api.cluster.local). If unreachable, the issue is upstream.
• Checking the node’s firewall or security group rules that may block port 6443.
• Engaging the cluster administrator to confirm that the node’s IP is within the allowed CIDR set.

Verification After Fix

1. Node health

   talosctl health --node <node-ip>
   

2. Kubernetes status

   kubectl get nodes -o wide
   

   The node should now show Ready with a Ready condition.

3. Log sanity check

   talosctl logs kubelet --node <node-ip> --tail 50
   

   Ensure no Failed or Error entries remain.

Cautions

• Reset wipes data: talosctl reset deletes all local data, including persistent volumes on the node. Use only if the node is not part of a critical workload.
• Disk path errors: Applying a config that points to a non‑existent device can render the node unbootable. Double‑check device identifiers before applying.

Practical Checklist

Use the following checklist before and after applying a new machine config to minimize downtime.

Item	Done?
Backup current machine config	✔
Validate CIDR, gateway, DNS	✔
Confirm disk device paths	✔
Ensure clusterEndpoint is correct	✔
Apply config and monitor talosctl health	✔
Verify node appears Ready in kubectl get nodes	✔

Conclusion

By following this structured diagnostic guide, most NotReady nodes can be restored with minimal disruption. The key is to isolate the failure point—network, disk, or endpoint—and apply the targeted fix before re‑checking health. Remember to back up configuration and be cautious with talosctl reset, as it will erase local data.