Managing State in Talos Linux: Ephemeral Root vs. Persistent Storage

The Immutability Conflict

Traditional Linux administration relies on the ability to modify /etc or /var to tune a system. In Talos Linux, this approach fails because the root filesystem is read-only. If you attempt to manually inject a configuration file into a system directory, that change will vanish upon the next reboot because /etc is implemented as a tmpfs (a temporary filesystem stored in volatile memory).

The core engineering challenge is deciding where to place configuration and data so that it survives a reboot without compromising the security and consistency of the immutable OS. The takeaway: Never attempt to modify the local filesystem for persistence; instead, define all state in the Machine Configuration YAML or via dedicated persistent mounts.

Storage Strategy Comparison

Depending on whether you are configuring the OS, the Kubernetes control plane, or an application, you must choose a different persistence mechanism.

Engineering Trade-offs

Choosing between the Machine Configuration and external persistent storage involves a trade-off between centralized truth and runtime flexibility.

• Machine Configuration: Ensures that every node in a cluster is identical. It eliminates configuration drift, where nodes diverge over time due to manual tweaks. However, changes often require a reboot or a specific API trigger to apply, increasing the time to implement a fix.
• Persistent Mounts: Allows applications to maintain state across restarts. While flexible, it introduces the risk of boot failures if a mount point is misconfigured or a disk fails to attach, as the OS may hang waiting for a critical path.

Implementing Persistent Configuration

To ensure a specific configuration or data path persists, you must define it in the machine configuration file before applying it to the node. This prevents the OS from reverting to the default read-only image state.

Example: Adding a persistent mount for system-level data

Run the following command from your local workstation (where talosctl is installed) to edit the configuration of a specific node. You will need admin permissions for the target node.

# Edit the machine configuration for a specific node
talosctl edit machineconfig -n <node-ip>

In the YAML editor, add a disk configuration to ensure a specific partition is mounted for your data, rather than relying on the ephemeral root:

machine:
  disks:
    - device: /dev/sdb
      partitions:
        - mountPoint: /var/lib/my-app-data

Risk: If you specify a device (e.g., /dev/sdb) that does not exist on the physical hardware, the node may fail to boot or enter a crash-loop as it attempts to mount a non-existent disk.

Validating the Immutability

To verify that your system is behaving as an immutable OS and that your persistence is working, perform the following checks:

1. Test Ephemerality: Use talosctl write to place a file in /etc. Reboot the node using talosctl reboot. Attempt to read the file; it should be gone.
2. Verify Config State: Run talosctl get machineconfig -n <node-ip>. Compare the output to your local YAML file to ensure the intended state is active.
3. Check Mounts: Use talosctl containers or inspect the node's logs to confirm that the persistent mount point (e.g., /var/lib/my-app-data) is active and writable.

Rollback Procedure

If a machine configuration change causes instability or boot failure, you must revert the configuration to the last known good state using the API:

# Apply a previous version of the config file
talosctl apply-config -n <node-ip> --file machine-config-backup.yaml