Erlang/OTP Node Distribution Failures: Diagnostic Guide for nodedown and Timeouts

Recognizable condition: a node is unreachable in the cluster

A healthy Erlang cluster shows peers in nodes() and net_kernel:connect_node/1 returns true. The failure pattern is nodedown reports, an empty or partial nodes() list, and {error,timeout} from net_kernel:connect_node/1 or net_adm:ping/1. The node may start, but distribution never completes.

The useful takeaway is to treat distribution as three independent gates: name resolution, EPMD reachability, and cookie authentication. If any gate fails, the connection stalls with a timeout rather than a clear error.

Cause to symptom table

Ordered checks

1. Verify cookie equality. Run in each node's Erlang shell: erlang:get_cookie(). Compare values. Required permissions: node access. Risk: printing cookie exposes secret.

2. Confirm EPMD is running and visible. On the host, run epmd -names. Expected check: the node name appears with a port. If not, EPMD is down or the node did not register. Requires local shell access.

3. Test name resolution and format. Use net_adm:ping('node2@host'). If you started with -sname, the name must be short and hostname must resolve via /etc/hosts or DNS. Risk: using -name with short hostname causes resolution failure.

4. Test TCP reachability. From the client host, test TCP 4369 to the peer host with netcat or telnet. Then test the distribution port range advertised by EPMD, commonly 9000-9100. Requires network access. Blocking here produces timeouts after EPMD lookup.

5. Review firewall and SELinux. Check iptables/nftables rules and SELinux booleans for port 4369 and distribution range. Requires root. Changes require restart of nodes to re-register.

6. Inspect logs. Enable logger for net_kernel and look for connection hints. Risk: debug logging increases volume.

7. Check OTP compatibility. Run erlang:system_info(otp_release) on each node. Distribution improvements are stable from R16 onward; TLS distribution requires compatible releases. Mismatch can cause silent authentication failures.

Fixes tied to findings

Cookie mismatch: set an identical cookie via a shared .erlang.cookie file with owner root and mode 400, or temporarily with erlang:set_cookie(Node, Cookie). Restart nodes after changing the file. Do not reuse a cookie across unrelated clusters.

EPMD unreachable: open TCP 4369 inbound on the peer host and ensure epmd is started by init/systemd. If you changed EPMD port, restart all nodes in the cluster; rolling updates cause temporary partitions.

Name resolution issue: use -name with fully qualified domain name or -sname with short name consistently. Ensure the hostname resolves both ways.

Firewall block: open TCP 4369 and the distribution port range. Verify with a TCP connect test before restarting nodes.

Duplicate name: rename the conflicting node and restart it. Check epmd -names to confirm uniqueness.

Version incompatibility: upgrade to a compatible OTP release, e.g., 25.x, and enable TLS distribution only when both nodes support it.

Escalation criteria

If nodes() remains empty after completing the checks, capture net_kernel diagnostics. Set net_kernel:setopts([{net_ticktime,60}]) and raise log level for net_kernel to debug. Collect crash dumps and EPMD listings from all hosts. Escalate to Erlang/OTP support or the OS/network team with persistent timeout or authentication failure details.

Limitations and practical verification

This guide covers classic TCP distribution. It does not cover TLS distribution certificate issues or container network overlays. A practical way to check result is to run net_adm:ping(Peer) and then nodes() on the local node. Successful resolution returns pong and the peer appears in the nodes list. Changes to cookies, ports, or node names require node restarts and may cause brief partitions.