YARN Resource Manager HA: Minimal Design and State Boundary Management

YARN Resource Manager High Availability: The Minimal Production Design

The primary operational risk in a single-ResourceManager (RM) YARN setup is complete cluster unavailability upon RM failure. The useful takeaway is that minimal production HA requires two RM hosts, a ZooKeeper quorum of three nodes, and client configuration that lists all RM logical addresses.

Core Architectural Components

• Active/Standby RMs: Two RM instances run, with one actively managing resources and the other passively waiting.
• ZooKeeper: Essential for leader election and state synchronization between the RMs. A quorum of three nodes prevents split-brain scenarios.
• Client Configuration: Clients must be configured with the full list of RM logical addresses, not just the current active one, to facilitate automatic failover.

Trust and Data Boundaries

Trust boundaries dictate where security enforcement is applied. The RM-to-NodeManager Remote Procedure Call (RPC) is a key boundary, often secured via Kerberos. However, Kerberos alone does not guarantee end-to-end encryption for data paths or container logs.

Concrete Example: A configuration enabling HA requires setting the RM address list (e.g., yarn.resourcemanager.ha.rm-addresses) to include both the active and standby host addresses, alongside the necessary ZooKeeper connection string.

Operational Verification and Failure Modes

Verification involves confirming the failover mechanism functions as designed, not just that the standby exists.

• Check: Inspect the RM web UI and Failover Controller logs to confirm the correct RM is elected as active post-failure.
• Failure Mode: Loss of the ZooKeeper quorum halts all RM activity, regardless of RM host health.
• State Risk: Standby lag means the standby might hold stale queue or application state, leading to application inconsistencies upon takeover.

Design Evolution Triggers

The minimal design must change when operational scope expands. Specifically, multi-tenancy requiring strict isolation mandates implementing queue-level Access Control Lists (ACLs) in addition to basic resource guarantees.

Limitation: RM HA does not inherently solve NodeManager resilience or container failure; YARN rescheduling handles those events.

Practical Check: Before and after simulating an RM shutdown, submit a test application. Confirm the client successfully reconnects to the new active RM and NodeManagers re-register without manual intervention.