What default memory-overutilization threshold should YARN administrators use for alerts?

Recommended Alert Thresholds

For a balanced alerting strategy that captures genuine risks while minimizing noise, YARN administrators should implement a two-tiered threshold system based on physical memory (pmem) utilization:

• Warning Threshold: 80%. This serves as an early indicator that a container is approaching its limit, allowing for proactive investigation before the NodeManager intervenes.
• Critical Threshold: 90% to 95%. This indicates an imminent risk of the container being killed by the YARN NodeManager for exceeding its memory limits.

Operational Context and Logic

YARN does not prescribe a universal default because memory usage patterns vary by workload (e.g., Spark vs. MapReduce). The 80/90 split is a common industry baseline designed to account for the following behaviors:

• JVM Spikes: Java-based applications often experience temporary memory spikes during garbage collection or heavy shuffle phases. A threshold below 80% frequently triggers false positives during these normal cycles.
• NodeManager Enforcement: The NodeManager monitors physical memory usage. If yarn.nodemanager.pmem-check-enabled is set to true, YARN will terminate containers that exceed their allocated limit. Alerts must trigger before this hard limit is hit to be actionable.

Verification and Tuning Steps

To verify that your chosen thresholds reduce false positives without missing critical events, follow these steps:

1. Audit Historical Logs: Search NodeManager logs for the specific error: "Physical memory usage is over the limit". Correlate the timestamps of these kills with your monitoring tool's memory metrics to see if the 80% warning triggered sufficiently in advance.
2. Baseline Normal Spikes: Monitor a representative set of stable jobs. If the "normal" peak usage consistently hits 85%, raise the Warning threshold to 88% to avoid alert fatigue.
3. Verify Configuration: Check your yarn-site.xml to ensure physical memory checks are active:

   <property>
     <name>yarn.nodemanager.pmem-check-enabled</name>
     <value>true</value>
   </property>

Required Diagnostic Detail

To refine this recommendation, please specify if you are monitoring Physical Memory (pmem) or Virtual Memory (vmem). Virtual memory checks are frequently disabled in Linux environments due to high false-positive rates and require entirely different threshold logic.