Cluster join_timeout and partition-induced formation failures in Neo4j causal clustering

Answer

A rejoining node invalidates the current leader only when its logical term (or epoch) is higher than the leader’s term. This happens if the node missed the leader’s election and its own election timeout fired before the partition healed, causing it to start a new election with a higher term. If the healed partition restores communication before any node’s election timeout expires, the existing leader retains authority.

Confirmed facts

• The default cluster.join_timeout is 60 seconds; if a node does not receive enough peer responses to reach quorum within this window, formation aborts and the log contains the message Cluster formation failed: timeout waiting for initial hosts.
• The default cluster.election_timeout is 10 seconds; it controls how quickly a node suspects lack of leadership and starts a new election.
• Leader validity is determined by comparing terms/epochs, not wall‑clock timestamps.

Likely explanation

1. During a partition, a minority of nodes may not hear from a majority; their join_timeout expires, they log the formation‑failure message and consider themselves unable to join.
2. If the partition heals before any node’s election_timeout elapses, the minority nodes still see the original leader’s term and step down as followers.
3. If the heal occurs after a node’s election_timeout has fired, that node increments its term, starts an election, and may win if its term is higher than the leader’s.
4. The side that first experiences an election timeout drives the new election; the winner becomes the leader only if its term exceeds the incumbent’s.

Verification steps

• Search neo4j.log for Cluster formation failed: timeout waiting for initial hosts and note the surrounding timestamps; compare them to the configured cluster.join_timeout value.
• After the partition heals, look for log lines such as New term X elected or Stepping down as follower due to higher term Y to see whether a rejoining node triggered a term increase.
• To test the influence of cluster.election_timeout, temporarily set it to a lower (e.g., 5 s) or higher (e.g., 20 s) value in neo4j.conf, repeat the partition‑heal scenario, and observe whether the leader changes according to the timeout relative to the healing interval.

Missing diagnostic detail

Please provide the term/epoch values shown in the log entries around the heal event (e.g., New term 3 elected or Current term 2) so we can confirm which side held the higher term and thus whether the leader was invalidated.