HDFS High Availability: Configuration & Troubleshooting Guide

What You Need to Know About HDFS HA

HDFS High Availability lets a cluster survive a NameNode failure without manual intervention. The key idea is that two NameNodes share a journal (via JournalNodes) and use ZooKeeper to elect which one is active. If the active dies, the standby steps up automatically and clients reconnect through the HA client resolver.

Core Configuration Steps

Below is a minimal core-site.xml that works on Hadoop 3.x. Replace placeholders with your values.

<configuration>
  <!-- Enable HA and specify the nameservice -->
  <property>
    <name>dfs.nameservices</name>
    <value>mycluster</value>
  </property>

  <!-- List the two namenodes -->
  <property>
    <name>dfs.ha.namenodes.mycluster</name>
    <value>nn1,nn2</value>
  </property>

  <!-- RPC addresses -->
  <property>
    <name>dfs.namenode.rpc-address.mycluster.nn1</name>
    <value>nn1-host:8020</value>
  </property>
  <property>
    <name>dfs.namenode.rpc-address.mycluster.nn2</name>
    <value>nn2-host:8020</value>
  </property>

  <!-- HTTP (WebHDFS) addresses -->
  <property>
    <name>dfs.namenode.http-address.mycluster.nn1</name>
    <value>nn1-host:9870</value>
  </property>
  <property>
    <name>dfs.namenode.http-address.mycluster.nn2</name>
    <value>nn2-host:9870</value>
  </property>

  <!-- JournalNode cluster -->
  <property>
    <name>dfs.namenode.shared.edits.dir</name>
    <value>journal://jn1:8485,journal://jn2:8485,journal://jn3:8485/;mycluster</value>
  </property>

  <!-- Enable HA client resolver -->
  <property>
    <name>dfs.client.failover.proxy.provider.mycluster</name>
    <value>org.apache.hadoop.hdfs.server.namenode.ha.FailoverProxyProvider</value>
  </property>
</configuration>

Run this file on every node (NameNode, JournalNode, DataNode, and client hosts). The dfs.namenode.shared.edits.dir points to a JournalNode cluster; the JournalNodes must be started before the NameNodes.

Operational Mechanics

• Election: ZooKeeper watches the dfs.ha.automatic-failover.enabled flag (default true). The active advertises itself; the standby follows the state of ZooKeeper. Only one NameNode can be active at a time.
• Log Replication: The active writes edits to the JournalNodes. The standby streams the same journal, keeping its namespace in sync.
• Client Reconnection: Clients use the dfs.client.failover.proxy.provider to resolve the current active. When the active changes, the provider automatically reconnects to the new RPC endpoint.

How to Verify HA Is Working

1. Start the cluster with both NameNodes up. Use hdfs dfsadmin -report to confirm both are in Active or Standby state.
2. Stop the active (e.g., service hadoop-namenode stop on the active host). Watch ZooKeeper logs; the standby should acquire the lock and become active.
3. Run a client command such as hdfs dfs -ls / from any node. The command should succeed without manual reconfiguration. Check the client log for a message like Switching to new active namenode nn2-host:8020.
4. Optionally, use hdfs dfsadmin -getServiceState to confirm the active state after failover.

Common Pitfalls and How to Avoid Them

• Shared storage on a local filesystem: If dfs.namenode.shared.edits.dir points to a local disk, a node restart will lose edits. Always use NFS, SMB, or a dedicated JournalNode cluster.
• ZooKeeper quorum misconfiguration: A quorum of 2n+1 ZooKeeper servers is required. With a 3-node cluster, use 3 ZooKeeper instances. If the quorum splits, you may see split‑brain and data corruption.
• Missing HA client provider: Clients that do not set dfs.client.failover.proxy.provider will continue to talk to the old active and fail after a failover. Add the property to core-site.xml on all client machines.
• Version mismatch: Hadoop 3.x HA uses JournalNodes; older 2.x clusters used a single shared edits directory. Mixing configurations across versions will fail.
• Insufficient JournalNode resources: Under‑provisioned JournalNodes can backlog edits during a long active period, delaying failover. Monitor journalnode logs for “write backlog” warnings.

Limitations of HDFS HA

HDFS HA does not eliminate the need for a SecondaryNameNode for checkpointing. The SecondaryNameNode still performs the merge of the checkpoint and edits files, keeping the namespace file small. However, it is not a failover candidate; if the active NameNode crashes, the SecondaryNameNode cannot assume its role.

HA also does not protect against DataNode failures or network partitions that isolate a NameNode from ZooKeeper. In such cases, the cluster may become read‑only until connectivity is restored.

Conclusion

By enabling HA and correctly configuring ZooKeeper, JournalNodes, and client properties, you can build an HDFS cluster that survives a NameNode outage with minimal downtime. The key is to treat the shared edits directory and ZooKeeper quorum as critical infrastructure; misconfiguring either can bring the whole cluster down.