Vault Raft retry_join nodes stay standalone in production HA

Recommended retry_join Method for Production

The most appropriate retry_join method depends entirely on your infrastructure:

• Cloud provider auto-join (AWS, GCP, Azure) — simplest when running in those environments. Uses instance metadata APIs and requires appropriate IAM permissions.
• Consul-based discovery — best if you already operate a healthy Consul cluster. Vault agents register as services; Consul handles DNS/health checks.
• Static leader_api_addr list — suitable for on-premises, air-gapped, or mixed environments. Requires manual updates when scaling.

Rule: Do not mix methods across nodes in the same cluster. All nodes must use the same discovery mechanism.

Why Nodes Report standalone Instead of Active/Standby

Nodes remain in standalone mode when they cannot form a Raft quorum. Confirmed causes include:

1. Insufficient simultaneous nodes — Production requires at least three nodes starting concurrently (or joined via vault operator raft join after the first node is active). Starting nodes sequentially with long delays lets the first node form a single-node cluster that rejects later joiners.
2. Unreachable cluster_addr / api_addr — cluster_addr must be an IP/hostname reachable by other Vault nodes (never localhost). api_addr must be reachable by clients. Mismatched addresses (e.g., private IP for cluster, public LB for API) are a frequent cause of join failures.
3. Port 8201 blocked — Bidirectional TCP 8201 must be open between all cluster nodes. Security groups, NACLs, and host firewalls must allow this traffic, not just client-to-active-node access.
4. Divergent storage.raft configuration — All nodes must have identical storage.raft.path and consistent retry_join blocks pointing to the same cluster.
5. TLS inconsistency — If tls_disable=0, certificate SANs must include the cluster_addr values used by peers. Mixed TLS settings across nodes prevent cluster formation.
6. DNS resolution failure at bootstrap — If retry_join uses hostnames, DNS or /etc/hosts must resolve before Vault starts.

Verification Steps

1. Check Vault logs for raft: initial configuration, cluster: joining, or repeated raft: node is in standalone mode messages.
2. Run vault operator raft list-peers on each node. Standalone nodes show only themselves.
3. Test connectivity: nc -zv <peer-ip> 8201 from each node to all others.
4. Confirm identical Raft config via vault read sys/storage/raft/configuration (requires unsealed active node).
5. Validate retry_join resolution: for cloud auto-join, verify IAM permissions with provider CLI; for Consul, check consul catalog services includes vault; for static list, curl each leader_api_addr/v1/sys/health.
6. After fixing configuration, restart all nodes simultaneously and monitor vault status on each. Expect one active, the rest standby (or performance standby in 1.12+).

One Diagnostic Detail Needed

Which infrastructure are you deploying on (AWS, GCP, Azure, on-premises, Kubernetes, or mixed)? This determines the recommended retry_join method and the specific IAM/security-group checks to prioritize.