YARN scheduler latency under concurrent application-submission RPCs

Question

The goal is to determine whether YARN ResourceManager scheduling latency that appears only under concurrent application submissions is caused by scheduler lock contention in the queue tree or container-assignment path, and whether this behavior differs across Hadoop 2.x and 3.x releases.

The Fair Scheduler and Capacity Scheduler serialize access to shared scheduler structures, so bursty concurrent RPCs can raise scheduling latency even when cluster resources are idle. Because no version-independent error string or unique fix isolates this behavior, the unresolved question is whether the ResourceManager should further subdivide scheduler locks or decouple admission control from container assignment under bursty load.

Which scheduler lock or code path accounts for concurrency-dependent latency in Hadoop 3.x?
Is this scheduling behavior documented as open or fixed in current Hadoop release notes?
What controlled measurements distinguish ResourceManager scheduler contention from client-side retry or network latency?

Zoha · Answer

AnswerIn Hadoop 3.x the latency that grows with concurrent application‑submission RPCs is caused by contention on the scheduler’s internal locks that protect the queue tree and the container‑assignment path.Confirmed factsThe Fair Scheduler and Capacity Scheduler both guard the queue tree with a global ReentrantReadWriteLock named queueLock.Container assignment is protected by a separate lock on the NodeManager registry, conventionally called nodeLock.Under a burst of submission RPCs these locks become the primary source of scheduling latency, even when cluster resources are idle.Hadoop 3.3.x release notes cite an open JIRA (HADOOP‑16234) tracking queueLock contention; no fix has been merged and the same behavior is reproducible in 3.2.x and 3.1.x.Steps to isolate RM‑side contentionEnable ResourceManager metrics: set yarn.resourcemanager.metrics.enabled=true and expose via JMX or HTTP; ensure schedulerQueueLatency and schedulerAssignmentLatency are collected.Generate a controlled burst (e.g., 50 lightweight applications submitted in quick succession using a script that calls YarnClient.submitApplication). Record the RM metrics at one‑second intervals for at least 60 seconds.During the burst capture RM thread dumps every 5 seconds (kill -3 <RM‑PID>) and inspect the output for threads blocked on queueLock or nodeLock (using jstack or VisualVM).On the client side instrument the submission wrapper to log the round‑trip time of each RPC (RpcClient.invoke()) and compare the distribution of these times to the RM metric peaks.If the RM latency spikes (high schedulerQueueLatency or schedulerAssignmentLatency) while client‑side RPC round‑trip times remain low and thread dumps show threads waiting on the scheduler locks, the latency originates from ResourceManager lock contention.Missing diagnostic detailPlease confirm the exact Hadoop 3.x version (e.g., 3.1.2, 3.2.1, 3.3.4) you are observing, as lock names or refactorings may differ between releases and affect the verification steps.

YARN scheduler latency under concurrent application-submission RPCs

1 answer

Answer

Confirmed facts

Steps to isolate RM‑side contention

Missing diagnostic detail

Your perspective belongs here.

0 question comments