YARN scheduler latency under concurrent application-submission RPCs
0 reputation · 04 Nov 2024, 16:42 UTC
The goal is to determine whether YARN ResourceManager scheduling latency that appears only under concurrent application submissions is caused by scheduler lock contention in the queue tree or container-assignment path, and whether this behavior differs across Hadoop 2.x and 3.x releases.
The Fair Scheduler and Capacity Scheduler serialize access to shared scheduler structures, so bursty concurrent RPCs can raise scheduling latency even when cluster resources are idle. Because no version-independent error string or unique fix isolates this behavior, the unresolved question is whether the ResourceManager should further subdivide scheduler locks or decouple admission control from container assignment under bursty load.
- Which scheduler lock or code path accounts for concurrency-dependent latency in Hadoop 3.x?
- Is this scheduling behavior documented as open or fixed in current Hadoop release notes?
- What controlled measurements distinguish ResourceManager scheduler contention from client-side retry or network latency?