YARN ResourceManager HA Failover Not Working: A Diagnostic Guide

The recognizable condition

Clients and applications report "ResourceManager not available" while the cluster appears up. One ResourceManager shows Active in its logs but does not serve RPCs, or both nodes stay in Standby and no leader is elected. The standby never promotes after the active is stopped or becomes unresponsive. This is a failover failure, not a simple RM crash.

Useful takeaway: In YARN HA, failover requires three coordinated pieces: ZooKeeper leader election, JournalNode shared state, and correct HA configuration. A failure in any one prevents promotion.

Cause to symptom table

Ordered checks

1. Confirm HA is configured on both nodes

Run on each RM host as the yarn user. Check core-site.xml and yarn-site.xml.

grep -E "yarn.resourcemanager.ha.enabled|yarn.resourcemanager.ha.rm-ids|yarn.resourcemanager.hostname" $HADOOP_CONF_DIR/yarn-site.xml

Expected: yarn.resourcemanager.ha.enabled true, two rm-ids defined, each host has a unique yarn.resourcemanager.hostname and matching RPC address. Risk: editing only one node splits the cluster.

2. Check service state via YARN admin

Run from an edge node with yarn client access.

yarn rmadmin -getServiceState rm1

Replace rm1 with the configured rm-id, e.g., rm1 or rm2. Expected output is ACTIVE or STANDBY. If command hangs or returns connection refused, the RM RPC is not reachable.

3. Inspect ZooKeeper leader election znode

Run on a ZooKeeper client host with zkCli.sh. Requires read access to ZooKeeper.

zkCli.sh -server zk1:2181 get /yarn-leader-election/

Expected: a znode value containing the hostname of the active RM. If node is missing or value is empty, election failed. Risk: manual znode edits can corrupt election.

4. Verify JournalNode health and edit log sync

Check JournalNode logs for SyncException and compare edit log indexes.

grep -i SyncException $HADOOP_LOG_DIR/journallogs/*

On active RM host:

cat $HADOOP_YARN_HOME/logs/yarn-*-resourcemanager-*.log | grep "Transitioning to active"

On standby RM host check for "Successfully recovered" and last applied transaction id. Mismatched indexes indicate lag.

5. Network and process checks

Confirm RM process is listening on configured RPC port, typically 8032.

ss -ltnp | grep 8032

Check ZooKeeper connectivity from both RM hosts:

echo stat | nc zk1 2181

Loss of connectivity to a majority of ZooKeeper quorum prevents election.

Fixes tied to findings

RM not responding but election intact. Do not kill the active RM abruptly. First verify standby is synchronized by checking JournalNode edit log index matches. If synchronized, gracefully stop the active RM via systemd or init script as yarn user. The ActiveStandbyElector should promote standby within seconds. If promotion does not occur, restart the standby RM service.

ZooKeeper unreachable. Restore network path and ZooKeeper quorum health. Ensure yarn.resourcemanager.zk-address points to a majority of the quorum. After quorum recovers, restart both RM services in rolling order: stop standby, start standby, stop active, start active. Changing HA configuration requires a rolling restart of all RM and JournalNode services; applying changes to only a subset can split the cluster.

JournalNode sync lag or state mismatch. Stop both RMs. Verify all JournalNodes are up and have consistent namespace. Clear any stale edits only under guidance from cluster operations. Restart JournalNodes first, then RMs. Do not stop the active RM abruptly without ensuring a standby is synchronized; doing so may cause loss of application state.

HA disabled or misconfigured. Correct yarn.resourcemanager.ha.enabled, yarn.resourcemanager.ha.rm-ids, yarn.resourcemanager.hostname, and RPC addresses in yarn-site.xml on all hosts. Apply changes consistently, then perform a rolling restart of RMs and JournalNodes.

Verification after corrective action

• Run yarn rmadmin -getServiceState rm1 and yarn rmadmin -getServiceState rm2 to confirm one ACTIVE and one STANDBY.
• Check ZooKeeper znode /yarn-leader-election/ for a valid leader value matching the active RM hostname.
• Verify JournalNode logs show no SyncException and that the active RM's edit log index matches the standby's recovered index.

Escalation criteria

Escalate if after rolling restarts both RMs remain STANDBY, ZooKeeper znodes are repeatedly created and deleted, or JournalNodes report persistent SyncException. Also escalate if application state is lost after failover, indicating checkpoint corruption. Collect RM logs, JournalNode logs, and ZooKeeper logs before escalation.

Limitations

This guide assumes YARN HA with ZooKeeper and JournalNodes is already enabled. It does not cover NFS-based shared storage or non-standard failover controllers. Version-specific log messages vary. Always verify configuration names against your distribution's documentation.