YARN Capacity Scheduler Preemption Not Reclaiming Containers: Diagnostic Guide

Problem and takeaway

A high-priority application remains in ACCEPTED while cluster utilization is near 100% and lower-priority queues continue running containers. The useful takeaway is that YARN Capacity Scheduler preemption is disabled by default in many distributions, and even when enabled it only acts after the ResourceManager confirms that a queue exceeds its maximum capacity and that a preemption target exists. If those conditions are not met, adding priority to an application will not free resources.

Recognizable condition

Look for this combination:

• High-priority queue application pending for several minutes.
• Cluster-wide memory or vCPU allocation close to capacity.
• Lower-priority containers running on nodes that match the pending request.
• ResourceManager logs contain no preemption-related events during the wait period.

Cause and diagnostic table

Observation	Likely cause
yarn.scheduler.capacity.preemption.enabled is false	Preemption is turned off; the scheduler never evaluates container reclamation.
Queue usage below maximum-capacity	No over-capacity condition, so preemption is not triggered.
Node labels do not match pending request	The scheduler cannot find a preemptable container on an eligible labeled node.
natural-termination-timeout has not elapsed	Preemption is deferred to let containers finish gracefully.
Repeated preemption requests ignored	The max-ignored-preemption-requests threshold was reached and the scheduler backed off.

Ordered checks

1. Confirm preemption is enabled. Run on the ResourceManager host with read access to the Hadoop configuration directory:

   grep -R "yarn.scheduler.capacity.preemption.enabled" /etc/hadoop/conf/capacity-scheduler.xml

   Expected result: <value>true</value>. If the property is absent or false, preemption is disabled. This is a read-only check with no cluster risk.

2. Compare queue usage against maximum capacity. Run on a client or edge node with the YARN client installed and permissions to query queues:

   yarn queue -status <queue-name>

   Replace <queue-name> with the high-priority queue path, for example root.production. Expected result: the queue's used capacity is at or above its maximum-capacity. If usage is below maximum, preemption will not start.

3. Inspect ResourceManager logs for preemption events. Run on the ResourceManager host with read access to the log directory:

   grep -i "preempt" $HADOOP_LOG_DIR/yarn-yarn-resourcemanager-*.log

   Expected result: lines containing PREEMTION, ContainerPreempted, or similar scheduler events. Absence of such lines during sustained contention suggests preemption is not evaluating or not finding targets.

4. Verify node labels and resource availability. Run on a client node:

   yarn node -list -all

   Expected result: nodes are in the expected RUNNING state and carry the labels required by the pending application. A label mismatch can cause preemption to be silently skipped.

5. Review preemption backoff and grace settings. Inspect capacity-scheduler.xml for:

   yarn.scheduler.capacity.preemption.max-ignored-preemption-requests
   yarn.scheduler.capacity.preemption.natural-termination-timeout

   If the ignored-requests threshold is low, the scheduler may stop attempting preemption after a few failures. If the natural termination timeout is long, containers receive an extended grace period before reclamation.

Fixes tied to findings

• Preemption disabled: set yarn.scheduler.capacity.preemption.enabled to true globally and, if needed, on the root queue using yarn.scheduler.capacity.root.preemption.enabled. Restart the ResourceManager to apply. Rollback: restore the previous XML values and restart the ResourceManager.
• Queue below maximum capacity: lower the queue's maximum-capacity so current usage exceeds it, or increase the pending application's resource request so the queue becomes over-allocated. Do not lower maximum capacity below the queue's guaranteed minimum without capacity planning.
• Node label mismatch: align node labels with the application's resource requirements, or remove unnecessary label constraints from the queue configuration.
• Grace period too long: reduce natural-termination-timeout in small increments. Watch for increased container churn; shorter timeouts reclaim resources faster but can interrupt applications more often.
• Backoff threshold too low: increase max-ignored-preemption-requests so the scheduler retries preemption instead of backing off.

Example configuration

This is a configuration example, not a verified deployment output:

<property>
  <name>yarn.scheduler.capacity.preemption.enabled</name>
  <value>true</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.preemption.enabled</name>
  <value>true</value>
</property>
<property>
  <name>yarn.scheduler.capacity.preemption.natural-termination-timeout</name>
  <value>60000</value>
</property>

Version assumption: Apache Hadoop YARN 3.x. Configuration names and defaults can differ by distribution.

Escalation criteria

Escalate to the platform team when:

• Preemption is enabled but no preemption events appear after 30 minutes of sustained contention.
• ResourceManager CPU or scheduler latency increases noticeably after enabling preemption, indicating container churn.
• A production SLA is at risk and queue tuning cannot be done safely during business hours.

Include the capacity-scheduler.xml excerpt, the last 200 lines of the ResourceManager log, and output from yarn queue -status for the affected queues.

Limitations and verification

Preemption does not guarantee immediate resource release. Containers receive a grace period, and an ApplicationMaster may decline to release containers. Aggressive preemption can increase ResourceManager overhead and reduce overall throughput.

Practical check after a change: submit a test job to the high-priority queue while a low-priority queue occupies the cluster. Confirm that yarn queue -status shows the high-priority queue using resources and that the ResourceManager log contains preemption events within the configured grace period.