YARN Capacity Scheduler: low-traffic cluster still billed for idle NodeManagers
0 reputation · 10 Aug 2021, 04:28 UTC
Goal
Reduce operating cost for a low-traffic Hadoop YARN workload. The cluster sits mostly idle between small, occasional jobs, and I want to confine that workload to a small guaranteed Capacity Scheduler queue while cutting the resources that are paid for but unused.
Constraints and uncertainty
Two documented levers seem relevant: shrinking the cluster through graceful NodeManager decommissioning (with a decommission timeout so running containers finish), or keeping the nodes and tightening queue capacity and maximum-capacity percentages so the workload cannot expand. Minimum-allocation memory and vcore settings may also force containers to reserve more than they use.
The unresolved decision is which lever is safer. NodeManagers are colocated with HDFS DataNodes, so removing nodes reduces storage capacity and risks under-replicated blocks if the cluster is not rebalanced first. YARN also has no built-in autoscaler, so any scale-down must come from the deployment environment. Assume a recent Hadoop 3.x release; exact property names should be confirmed against the deployed version.
Questions
- For a low-traffic workload, is decommissioning nodes or tightening queue capacity the safer first cost-reduction step when HDFS is colocated?
- How should I verify, via the ResourceManager UI, that queue capacities and container allocations reflect the change before and after?
- What HDFS replication checks should pass before considering a node removal complete?