Diagnosing HDFS NameNode HA Automatic Failover Failures in Hadoop 2.7+ and 3.x

Recognizable Condition

The active NameNode crashes or becomes partitioned, but the standby never takes over. ZKFC logs show Failed to elect active or Fencing failed. Both NameNodes remain in standby and the cluster enters an unsafe mode where no writes succeed.

Cause / Diagnostic Table

Ordered Checks

1. Verify ZooKeeper ensemble – Run on any ZooKeeper node (requires zkCli.sh in PATH):

   zkCli.sh -server zk1:2181,zk2:2181,zk3:2181 ls /hadoop-ha

   Expect a list containing ActiveStandbyElectorLock. If the command hangs or returns Node does not exist, quorum is lost.
2. Check ZKFC processes – On each NameNode host (as hdfs user):

   jps | grep ZKFC

   Missing process means ZKFC isn’t running; start with hdfs --daemon start zkfc.
3. Inspect ZKFC logs – Look in $HADOOP_LOG_DIR/hadoop-hdfs-zkfc-<host>.log for lines containing Electing active and Fencing. Note timestamps and error codes.
4. Test fencing command manually – On the standby NameNode (or a client with SSH keys):

   ssh nn1.example.com 'hdfs dfsadmin -fs hdfs://nn1:8020 -shutdownDatanode <dn-host>'

   Requires passwordless sudo for hdfs on the target. Exit code 0 confirms the fence works; non‑zero indicates mis‑configuration.
5. Confirm JournalNode quorum – On any JournalNode host:

   hdfs jn -status

   Output must show a majority (e.g., 3 of 3, 3 of 5). Bring up any down nodes and verify dfs.namenode.shared.edits.dir points to the correct quorum.
6. Validate NameNode RPC/HTTP accessibility – From each ZKFC host:

   telnet nn1.example.com 8020
    telnet nn1.example.com 9870
    telnet nn1.example.com 8485

   All three ports must accept connections. Firewall or security‑group rules blocking any of them will cause ZKFC health checks to fail.

Fixes Tied to Findings

• ZooKeeper quorum loss (1) – Restart the affected ZooKeeper nodes until a majority is online. Verify with zkCli.sh that the ActiveStandbyElectorLock znode reappears.
• Fencing mis‑configuration (2) – Edit hdfs-site.xml on both NameNodes:

  <property>
    <name>dfs.ha.fencing.methods</name>
    <value>shell(/opt/hadoop/bin/fence.sh)</value>
  </property>

  Ensure fence.sh exists, is executable, and uses passwordless SSH keys. Test again with step 4.
• Network partition (3) – Enable dfs.ha.automatic-failover.enabled=true in hdfs-site.xml and provision redundant network paths (bonded NICs, separate VLANs). Verify that each NameNode can reach the ZooKeeper ensemble over distinct routes.
• JournalNode quorum unavailable (4) – Start missing JournalNodes, then run hdfs namenode -recover on the active NN if edits are stale. Confirm dfs.namenode.shared.edits.dir lists a majority of JNs.
• Firewall blocks (5) – Open TCP ports 8020 (RPC), 9870 (HTTP), and 8485 (ZKFC) on all NameNode and ZKFC hosts. In cloud environments, adjust security groups accordingly.

Escalation Criteria

Engage the platform team when:

• Failover does not complete within dfs.ha.fencing.timeout.ms (default 10 s) after three consecutive automatic attempts.
• Both NameNodes report HAState=active simultaneously (split‑brain).
• Manual failover via hdfs haadmin -failover --forceactive <targetNN> is required.

Provide the team with ZKFC logs, ZooKeeper znode state, and the output of the ordered checks.

Limitations & Practical Verification

Automatic failover requires ZooKeeper 3.4.6+; older releases may not honor session expiration reliably. Fencing via SSH depends on passwordless sudo for the hdfs user—mis‑configured /etc/sudoers yields silent failures. Never disable dfs.ha.fencing.methods; without fencing, split‑brain can corrupt HDFS metadata. JournalNode quorum must be odd‑numbered (3, 5); losing majority halts edits and blocks failover. If the NameNode web UI is disabled (dfs.namenode.http-address=0.0.0.0:0), ZKFC cannot monitor the standby.

Verification steps (run after any fix):

1. Simulate failover: hdfs haadmin -failover nn1 nn2 – confirm active switches in the NameNode web UI and ZKFC logs.
2. Run the configured fence command against the current active NN; ensure it returns 0 and the NN transitions to standby.
3. Check that the znode /hadoop-ha/<nameservice>/ActiveStandbyElectorLock exists and its holder matches the active NN.
4. Monitor HA metrics (NameNode HAState, ZKFC election latency, JournalNode sync lag) via JMX or Prometheus exporters.