YARN HA with ZooKeeper – Minimal Design & Operational Guide

Problem Statement

When a YARN cluster grows beyond a handful of nodes, a single ResourceManager (RM) becomes a single point of failure. Clients must experience uninterrupted job execution even if the active RM crashes. The challenge is to orchestrate failover without losing cluster state, while preserving security boundaries and minimizing operational complexity.

Architectural Requirements

• Transparent client failover: jobs continue running, no resubmission needed.
• Consistent cluster view: all RMs must agree on the set of running containers and applications.
• Compatibility with existing HDFS High Availability and Kerberos security.
• Minimal operational footprint: no custom services beyond the RM, ZooKeeper quorum, and a shared state store.

Smallest Suitable Design

The canonical HA setup for YARN uses the following components:

• Two RM instances – one active, one standby. The standby never accepts client RPCs.
• ZooKeeper quorum (≥3 nodes) – provides leader election and fencing support.
• Shared state store – HDFS (default) or LevelDB, holds RM internal state (application records, container allocations).
• RPC proxy / client‑side failover library – redirects all client calls to the active RM. In Hadoop 3.x, this is built‑in via yarn.resourcemanager.webapp.address and yarn.resourcemanager.address configuration.
• Kerberos integration – secures RM‑to‑ZooKeeper, RM‑to‑state‑store, and client‑to‑RM communication.

Figure 1 illustrates the data flow: clients hit the proxy, which forwards RPCs to the active RM. The standby RM watches ZooKeeper for leadership changes and pulls state from the shared store when it becomes active.

Trust / Data Boundaries

• ZooKeeper is trusted only for leader election. All data stored in ZooKeeper must be minimal (e.g., election znode, fencing information).
• RPC Proxy is trusted by clients to present the active RM. It must not cache or modify requests.
• State Store (HDFS or LevelDB) is protected by ACLs and Kerberos. Only RMs can read/write the state file.
• Internal RM‑to‑RM communication (e.g., RMStateStore sync) must use mutual authentication to prevent tampering.

Operational Checks

1. ZooKeeper session health
   

   # Run on any ZooKeeper node
   $ bin/zkCli.sh -server zk1:2181 -server zk2:2181 -server zk3:2181 mntr
   # Look for 'zk_server_state=leader' or 'follower'
   

   Expected: one node reports leader, others follower. If all show standby or session expires, the quorum is broken.

2. RM Web UI
   Access http://<rm-host>:8088/cluster on each RM. The active RM shows State: ACTIVE; standby shows State: STANDBY.
3. Logs
   Check yarn.log for events: RMStateStore: Restoring state from HDFS, RM State: ACTIVE, and RM State: STANDBY. Any abnormal errors (e.g., Failed to acquire lock) indicate misconfiguration.
4. Automated failover test
   

   # Stop active RM abruptly
   $ kill -9 $(pgrep -f 'org.apache.hadoop.yarn.server.resourcemanager.ResourceManager')
   # Wait ~15s (default failover timeout)
   # Verify standby becomes active via Web UI or
   $ yarn rmadmin -getServiceState
   # Expected output: 'ACTIVE'
   

   Risk: kill -9 can leave orphaned containers; ensure yarn.resourcemanager.aux-services cleans up.

Failure Modes

• Split‑brain: If a RM is isolated from ZooKeeper but still believes it is active, it may continue allocating containers. The fencing mechanism (ZooKeeper's zooKeeper.fence) mitigates this by preventing two RMs from writing to the state store simultaneously.
• ZooKeeper quorum loss: All RMs lose connectivity to the quorum; leader election stalls, cluster stalls. Monitor mntr and set zookeeper.session.timeout appropriately.
• State store corruption: Corrupted application records in HDFS/LevelDB prevent RM startup. Enable yarn.resourcemanager.state-store.checksum and schedule regular backups.
• Misconfigured RPC proxy: Clients may keep sending requests to a dead RM if the proxy fails to refresh its list of active RMs. Ensure yarn.resourcemanager.webapp.address is correctly set and the proxy refresh interval is short (<30s).

Conditions That Prompt a Design Change

• Adopting Kubernetes or Mesos as the underlying container orchestrator – these platforms provide native leader election; the RM can be replaced by a lightweight controller.
• Moving to a restart‑less state recovery model – e.g., using RMStateStore with in‑memory snapshotting and less frequent persistence.
• Replacing ZooKeeper with etcd or Consul – for environments that already use these consensus engines, migrating the election mechanism can reduce operational overhead.
• Scaling to dozens of RMs – the two‑RM design may become a bottleneck; consider a multi‑RM cluster with a shared state store and a hierarchical leader election.

Configuration Example

Below is a minimal yarn-site.xml snippet for a two‑RM HA cluster. Replace placeholders with your environment values.

<configuration>
  <property>
    <name>yarn.resourcemanager.ha.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>yarn.resourcemanager.ha.rm-ids</name>
    <value>rm1,rm2</value>
  </property>
  <property>
    <name>yarn.resourcemanager.ha.zookeeper-quorum</name>
    <value>zk1:2181,zk2:2181,zk3:2181</value>
  </property>
  <property>
    <name>yarn.resourcemanager.ha.zk-session-timeout</name>
    <value>15000</value>
  </property>
  <property>
    <name>yarn.resourcemanager.state-store</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.RMStateStoreHDFS</value>
  </property>
  <property>
    <name>yarn.resourcemanager.state-store.hdfs.path</name>
    <value>/yarn/rmstate</value>
  </property>
  <property>
    <name>dfs.namenode.shared.edits.dir</name>
    <value>hdfs://nn1:8020/dfs/name/history</value>
  </property>
  <!-- Kerberos settings -->
  <property>
    <name>yarn.resourcemanager.keytab</name>
    <value>/etc/security/keytabs/rm.keytab</value>
  </property>
  <property>
    <name>yarn.resourcemanager.principal</name>
    <value>rm/[contact removed]</value>
  </property>
</configuration>

Practical Validation Steps

1. Deploy two RMs with the above config on separate hosts.
2. Verify ZooKeeper quorum is healthy: zkCli.sh mntr.
3. Submit a long‑running MapReduce job via yarn jar.
4. Kill the active RM process and watch the standby become active within 10–12 seconds.
5. Check the job continues: yarn application -status <app-id> should show RUNNING.
6. Inspect logs for any Failed to acquire lock or Session expired messages.

Conclusion

By limiting the HA design to two RMs, a ZooKeeper quorum, a shared state store, and Kerberos‑protected communication, you achieve transparent client failover with minimal operational overhead. Regular health checks, automated failover tests, and vigilant monitoring of ZooKeeper sessions are essential to detect and recover from the most common failure modes. When your cluster scales or your orchestration stack evolves, revisit the design to incorporate container‑native leader election or alternative consensus engines.