Designing a Minimal HDFS High Availability Setup with Quorum Journal Manager

Requirements for HDFS HA with Quorum Journal Manager

HDFS High Availability (HA) removes the single point of failure of the NameNode by running two NameNodes in an active/standby pair. The edit log, which records all namespace changes, must be replicated to a set of JournalNodes using the Quorum Journal Manager (QJM). To guarantee consensus, QJM requires an odd number of JournalNodes, with a minimum of three, so that a majority (quorum) can always be formed. The active NameNode writes each edit to a majority of JournalNodes; the standby NameNode continuously tails the same quorum to stay in sync. Clients continue to talk to the active NameNode for all namespace operations.

Trust and Data Boundaries

• JournalNode zone: JournalNodes and both NameNodes must reside in the same trusted network zone because they exchange edit log records over RPC. If an attacker can intercept or forge these messages, they could cause namespace divergence.
• Client trust: Clients trust the active NameNode for read and write operations. The standby NameNode only replays edits and does not serve client requests until it becomes active.
• Fencing boundary: When a failover occurs, the previously active NameNode must be fenced (e.g., via SSH or shell fencer) before the standby takes over. The fencing mechanism operates outside the HDFS trust zone and must be reliable; a failure here can leave both NameNodes active, leading to split‑brain.

The Smallest Suitable Design

Based on the research findings, the minimal HA cluster that satisfies all requirements consists of:

• 2 NameNodes (one active, one standby)
• 3 JournalNodes (odd count ≥3)
• A shared storage device for the NameNode’s fsimage (e.g., NFS, SAN, or a distributed block store). The edits directory is not shared; it is replicated via QJM.
• DataNodes unchanged from a non‑HA cluster.
• Optional but recommended: a ZooKeeper quorum (≥3 nodes) to run the automatic failover controller (ZKFC).

This layout avoids any single point of failure for the namespace while keeping operational overhead low. The shared fsimage storage must be highly available; if it becomes unavailable, both NameNodes cannot start, but the edit log remains safe on the JournalNodes.

Operational Checks

After deploying the cluster, verify the following on a regular basis:

1. JournalNode quorum health: Run hdfs haadmin -checkHealth on any NameNode. The command reports whether a majority of JournalNodes are reachable and in sync.
2. NameNode state: Use hdfs haadmin -getServiceState -nnId <namenode-id> to confirm one NameNode returns active and the other standby.
3. Edit log lag: Monitor the metric EditLogTailer#Lag via JMX or the NameNode UI. A steadily increasing lag indicates the standby is falling behind.
4. Fencing script logs: Check the NameNode logs for messages like Fencing succeeded or Fencing failed after a failover trigger.

These checks can be automated with monitoring tools (e.g., Ganglia, Prometheus) that scrape the JMX endpoints.

Failure Modes and Expected Behavior

• Loss of a single JournalNode: With three JournalNodes, losing one leaves two, which still constitutes a majority. Writes continue without interruption; the cluster remains in a degraded but safe state. If a second JournalNode fails, quorum is lost and the active NameNode will stall on edit log writes.
• Active NameNode crash: The standby NameNode, having replayed all available edits, will transition to active after the configured failover timeout (controlled by ha.zookeeper.failover-controller if ZooKeeper‑based failover is used). The previously active NameNode must be fenced before the standby assumes the role.
• Fencing misconfiguration: If the fencer does not successfully isolate the old active NameNode, both NameNodes may attempt to write to the JournalNodes simultaneously, causing edit log divergence and potential namespace corruption.
• Network partition isolating JournalNodes: If the active NameNode cannot reach a majority of JournalNodes, it will stop accepting writes and log errors about being unable to achieve quorum.

Conditions That Would Trigger a Redesign

Consider revisiting the HA design when any of the following arise:

• Write latency becomes unacceptable due to the round‑trip to JournalNodes. In latency‑sensitive workloads, you might evaluate a shared‑storage HA approach (e.g., using a highly available NAS) despite its own failure domains.
• Operational overhead of managing an odd number of JournalNodes and ensuring fencing reliability outweighs the benefits. Simpler architectures like a standby NameNode with shared storage might be preferable if the environment can guarantee the storage’s availability.
• Regulatory or security policies require stricter isolation of the edit log path. If JournalNodes cannot be placed in the same trusted zone as the NameNodes, you would need to adopt an alternative replication mechanism (e.g., HTTP‑based edit log transfer with mutual TLS).
• The cluster scales to thousands of NameNode pairs (e.g., a federation of independent namespaces). Managing many separate HA sets may become cumbersome, prompting a move to a centralized coordination service or a different HA pattern.

Practical Verification Steps

To confirm that your HA setup behaves as expected, perform the following checks in a test environment. All commands assume you have HDFS superuser privileges.

1. Validate initial state:

   # Check NameNode states
   hdfs haadmin -getServiceState -nnId nn1
   hdfs haadmin -getServiceState -nnId nn2
   # Should return active and standby respectively
   
   # Check JournalNode quorum
   hdfs haadmin -checkHealth
   

2. Simulate active NameNode failure:

   # Identify the active NN (e.g., nn1)
   ssh namenode1 "pkill -f NameNode"
   # Monitor logs on the standby for transition
   ssh namenode2 "tail -f /var/log/hadoop/hdfs/hdfs-namenode-*.log" | grep -i "Transitioning to active"
   

   After the failover, run hdfs haadmin -getServiceState again to confirm the former standby is now active.

3. Induce a JournalNode failure:

   # Stop one JournalNode (e.g., jn2)
   ssh journalnode2 "pkill -f JournalNode"
   # Verify writes still succeed
   hdfs dfs -put /etc/passwd /tmp/testfile
   # Check quorum health
   hdfs haadmin -checkHealth
   

   The command should report that a quorum is still maintained.

4. Check edit log durability across failover:

   # Before failover, create a file
   hdfs dfs -put /etc/hosts /tmp/before
   # Trigger failover as above
   # After failover, verify the file exists and can be read
   hdfs dfs -ls /tmp/before
   hdfs dfs -cat /tmp/before
   

If any of these steps fail, review the corresponding logs (NameNode, JournalNode, ZooKeeper if used) and verify network connectivity, time synchronization (NTP), and that the fencing script returns a non‑zero exit code only when it successfully isolates the target node.

Limitations

Even with a correct HA deployment, the following limitations remain:

• The edit log write path incurs additional network latency proportional to the round‑trip time to the JournalNode quorum. Measure this with a baseline test (e.g., hdfs dfs -put on a large file) before and after enabling HA.
• The shared fsimage storage becomes a single point of failure for NameNode startup. Ensure it is backed by a reliable replication mechanism (e.g., NFS with active‑passive failover or a distributed block store).
• Fencing must be tested under realistic failure scenarios (power loss, network split). A fencer that relies on SSH may fail if the target node’s SSH daemon is unresponsive.

Regularly exercising the failover path (e.g., quarterly drills) helps catch configuration drift before it impacts production.