Preventing Resource Starvation in Multi-Tenant YARN Clusters with Preemption

The Problem: Low-Priority Jobs Holding Resources Hostage

When running a shared Hadoop cluster, a single team's long-running batch job can unintentionally starve critical workloads from other teams. Without preemption, the Capacity Scheduler's guaranteed capacity percentages only work when queues have available resources to spare. Once a queue claims its full allocation, everyone downstream waits.

The Solution: Configure Queue Preemption

Preemption allows the Capacity Scheduler to forcibly relocate containers from lower-priority queues to satisfy higher-priority ones. This requires enabling the feature and defining which queues can reclaim resources from others.

Key Configuration Properties

Preemption operates through several interdependent settings in capacity-scheduler.xml:

• yarn.resourcemanager.scheduler.monitor.enable - Must be true to activate preemption monitors
• yarn.resourcemanager.monitor.enable - Enables the preemption policy logic
• yarn.scheduler.capacity.preemption.enabled - Master switch for preemption behavior
• yarn.scheduler.capacity.root.production.always-preempt-non-faithful - Forces preemption when production queue isn't fully satisfied

Worked Example: Production vs Development Queues

Consider a cluster with two queues: production (70% capacity) and development (30% capacity). A production Spark job needs 80 nodes, but development has already claimed its full 30% allocation.

Configuration in capacity-scheduler.xml:

<property>
  <name>yarn.scheduler.capacity.root.queues.preemption.disabled</name>
  <value>false</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.development.always-preempt-non-faithful</name>
  <value>false</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.production.always-preempt-non-faithful</name>
  <value>true</value>
</property>

<property>
  <name>yarn.scheduler.capacity.preemption.max-wait-time-before-kill-seconds</name>
  <value>300s</value>
</property>

<property>
  <name>yarn.scheduler.capacity.preemption.nocancelalaforexec-timeout-secs</name>
  <value>60s</value>
</property>

This configuration ensures production jobs can reclaim resources from development after a 5-minute grace period, preventing indefinite starvation while giving development jobs a chance to complete gracefully.

Trade-offs and Limitations

Preemption isn't free. Each preempted container requires a new allocation elsewhere, potentially causing job failures if the target queue can't accommodate the relocated work. The nocancelalaforexec-timeout-secs property controls how long a preempted container can continue executing before being killed, balancing between job stability and resource responsiveness.

Additionally, preemption only works when the dominant resource (typically the scarcest resource across queues) is over-allocated. In CPU-only environments using CapacitySiloCalculator, preemption decisions are simpler but may miss opportunities in multi-resource clusters.

Verification Steps

1. Confirm scheduler class: In yarn-site.xml, verify yarn.resourcemanager.scheduler.class is set to org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
2. Check preemption is active: Run yarn queue -status production and look for Preemption disabled: false
3. Monitor preemption decisions: Check ResourceManager logs for Preempting messages when high-priority jobs are submitted

Actionable Next Steps

Start by enabling preemption on a single parent queue rather than the entire hierarchy. Monitor job success rates and adjust the max-wait-time-before-kill-seconds based on your workload characteristics. For clusters running both CPU and memory intensive workloads, consider using DominantResourceCalculator to ensure preemption decisions account for both dimensions.