YARN Capacity Scheduler: low-traffic cluster still billed for idle NodeManagers

Answer

For a low‑traffic YARN workload where NodeManagers also run HDFS DataNodes, the safer first cost‑reduction step is to tighten the guaranteed and maximum capacity of the target Capacity Scheduler queue rather than decommissioning nodes.

Why queue tightening is safer

Decommissioning a NodeManager removes both compute and storage. If HDFS storage is near full or blocks are under‑replicated, removing a DataNode can cause under‑replicated blocks and potentially trigger namenode safemode. Tightening queue capacity leaves the NodeManager count unchanged, so HDFS replication and storage capacity stay intact while limiting the resources the workload can consume.

Confirmed facts (Hadoop 3.x)

• Queue properties yarn.scheduler.capacity.root.<queue>.capacity (guaranteed) and ...maximum-capacity control the share of cluster resources the queue may use.
• Changing these values does not affect the number of active NodeManagers.
• Minimum allocation settings (yarn.scheduler.minimum-allocation-mb and -vcores) can cause containers to reserve more than they actually use, but they do not force the scheduler to allocate more containers than the queue’s capacity permits.

Steps to tighten queue capacity

1. Open the ResourceManager UI → Scheduler → Queues.
2. Note the current guaranteed capacity, maximum capacity, used capacity and list of running containers for the target queue (this is the “before” snapshot).
3. Edit capacity-scheduler.xml (or use the RM’s REST API) to reduce the queue’s capacity and optionally maximum-capacity to the desired levels, e.g., from 100% to 20% guaranteed and 40% maximum.
4. Save the file and run yarn rmadmin -refreshQueues to reload the configuration without restarting the RM.
5. Return to the RM UI → Scheduler → Queues and verify that the queue’s configured capacity and maximum capacity now match the new values and that used capacity has dropped accordingly.
6. Confirm that the Active NodeManagers count under Nodes → Active NodeManagers is unchanged.

When to consider node decommissioning

If after tightening the queue the cluster still runs more NodeManagers than needed, decommissioning can be explored—but only after confirming that HDFS can tolerate the loss of storage.

Missing diagnostic detail that would change the recommendation

What is the current HDFS storage utilization percentage (or the number of under‑replicated blocks)? If utilization is high (> 80 %) or there are under‑replicated blocks, decommissioning is unsafe; if utilization is low and replication is healthy, decommissioning may be considered.

HDFS replication checks before removing a NodeManager

1. Run hdfs dfsadmin -report and verify:
   • Live DataNode count matches the expected number after removal.
   • Under Replicated Blocks = 0.
   • Total capacity aligns with remaining nodes.
2. Check the namenode logs for any safemode entry; ensure dfs.namenode.safemode.threshold-pct is not exceeded.
3. Optionally run hdfs fsck / -files -blocks -locations to confirm no corrupt or missing blocks.

Graceful decommission steps (if utilization is low)

1. Add the NodeManager host(s) to etc/hadoop/yarn.excludes (or the file pointed to by yarn.resourcemanager.nodes.exclude-path).
2. Set a sufficient timeout, e.g., yarn.resourcemanager.nodemanager-graceful-decommission-timeout-secs to a value longer than the longest expected container runtime.
3. Run yarn rmadmin -refreshNodes to start the decommission process.
4. Monitor the Nodes → Decommissioning NodeManagers list in the RM UI until the state changes to “Decommissioned”.
5. After decommission, repeat the HDFS replication checks above to confirm no under‑replicated blocks appeared.