Diagnosing and Fixing JVM Heap Pressure in Elasticsearch Nodes

Recognizing JVM Heap Pressure in Elasticsearch

When a node repeatedly hits high heap usage (>80%) and the cluster health shifts to yellow or red, the most common culprit is JVM heap pressure. Symptoms include:

• Frequent stop‑the‑world GC pauses in the logs (often >200 ms).
• High heap.percent on the /_cat/nodes endpoint.
• Thread‑pool queues for bulk or search exceeding the configured thresholds.
• Shard reallocation delays and increased node latency.

Below is a concise diagnostic guide that walks you through the checks, fixes, and when to involve Elastic support.

Cause & Diagnostic Table

Possible Cause	Diagnostic Indicator	Immediate Check
Heap size too large (>32 GB) causing long GC pauses	Heap.percent > 80% and long GC pause times	Run GET /_cat/nodes?v&h=heap.percent,heap.max,heap.used,heap.avail
Fielddata or doc values cache over‑consumption	Cache usage near or above indices.fielddata.cache.size	Check GET /_nodes/stats/jvm and GET /_nodes/stats/indices
Thread‑pool back‑pressure from bulk/search requests	Queue lengths > 75% of pool size	Run GET /_nodes/stats/thread_pool
Cluster health shifting to yellow/red after GC spikes	Reallocation delays in GET /_cluster/health	Monitor GET /_cluster/health?pretty

Ordered Checks

1. Verify Heap Utilization

   GET /_cat/nodes?v&h=heap.percent,heap.max,heap.used,heap.avail

   Look for nodes where heap.percent exceeds 80%. Note the heap.max value; if it is >32 GB, consider resizing.

2. Inspect GC Logs

   Enable gc.log or use jcmd VM.log to capture stop‑the‑world events. Search for pauses >200 ms.

3. Check Thread‑Pool Queues

   GET /_nodes/stats/thread_pool

   Identify pools (e.g., bulk, search) where queue is >75% of size. High queue lengths indicate back‑pressure.

4. Assess Cache Usage

   GET /_nodes/stats/indices

   Look at indices.fielddata.memory_size_in_bytes and indices.fielddata.evictions. Frequent evictions suggest the cache is too small or the fieldtype is inappropriate.

5. Cluster Health & Shard Reallocation

   GET /_cluster/health?pretty

   Check status, unassigned_shards, and delayed_unassigned_shards. Long delays correlate with GC activity.

Fixes Tied to Findings

• Resize JVM Heap

  Reduce -Xmx to 1/4 of physical RAM (max 32 GB). For example, change ES_JAVA_OPTS to -Xms16g -Xmx16g. Restart the node after the change.

• Adjust Fielddata/Doc Values Cache

  If fielddata usage is high, convert large text fields to keyword or enable doc_values where appropriate. Alternatively, increase indices.fielddata.cache.size but only if heap allows.

• Tune Thread‑Pool Sizes

  Increase bulk or search pool sizes via ES_THREAD_POOL_SIZE or cluster.thread_pool.bulk.size. Monitor queue lengths after each increment.

• Reduce Bulk Load Rate

  Throttle bulk ingestion by adjusting the producer rate or using bulk.flush.min_content_size. This reduces bursty GC.

• Optimize Mappings

  Remove unnecessary fielddata on text fields, disable norms if not needed, and use doc_values for numeric fields.

• Enable GC Logging & Tune GC Algorithm

  For Java 11+, switch to G1GC with -XX:+UseG1GC if not already. Verify that pause times drop below 200 ms.

Escalation Criteria

1. After applying all fixes, heap.percent remains >80% for more than 30 minutes.
2. GC pause times consistently exceed 500 ms despite heap resizing.
3. Cluster health stays yellow/red for >15 minutes and shard reallocation stalls.
4. Thread‑pool queues stay above 90% of capacity for >10 minutes.

At this point, gather the following and open a support ticket:

• Full JVM heap dump (jcmd VM.heap_dump).
• Recent GC logs (last 24 hours).
• Cluster configuration (GET /_cluster/settings, GET /_nodes/settings).
• Mapping and index settings for affected indices.

Limitations & Practical Verification

The guide assumes Elasticsearch 8.x running on Java 11+. Heap sizing rules change with newer Java versions, so verify the 1/4 rule against your Java documentation. Also, reducing heap may increase GC frequency; monitor gc.log after changes to confirm pause times improve, not just heap usage.

Use Kibana’s Monitoring UI or X-Pack metrics to continuously watch heap.percent, GC pause times, and thread‑pool queues. Set alerts for thresholds that match the escalation criteria.