Diagnosing Yellow/Red Cluster Health in Elasticsearch: Unassigned Shards, Sharding, and Replication

Recognizable Condition

A cluster reports yellow or red status in the /_cluster/health API and lists one or more unassigned_shards in the unassigned_shard array. The cluster is still reachable, but some data is not fully replicated or even stored.

Root‑Cause Table

Cause	Typical Symptoms
Primary shards > available data nodes	All nodes are data nodes; cluster still Yellow
Node offline or removed	Shard unassigned because its owner is gone
Insufficient disk space	Allocation explain shows disk watermarks denied
Allocation filtering or routing rules	Explain shows allocation filtering or routing constraints
High allocation delay	Shard waits > 30s in unassigned_shards

Ordered Diagnostic Checks

1. Cluster Health Snapshot

   curl -X GET "http://localhost:9200/_cluster/health?pretty"
   

   Verify status and unassigned_shards. Expected: green when healthy.

2. List Unassigned Shards

   curl -X GET "http://localhost:9200/_cat/shards?format=json&h=index,shard,prirep,state,unassigned.reason"
   

   Filter for UNASSIGNED rows to see which indices are affected.

3. Explain Allocation for a Specific Shard

   curl -X GET "http://localhost:9200/_cluster/allocation/explain?pretty" \
    -d '{"index":"my_index","shard":0}'
   

   Look for reason field: disk_watermark, node_left, allocation_filter, etc.

4. Check Disk Utilization on Data Nodes

   curl -X GET "http://localhost:9200/_cat/nodes?v&h=ip,disk.used_percent,disk.total,disk.used"
   

   Ensure used_percent is below cluster.routing.allocation.disk.watermark.high (default 90%).

5. Verify Allocation Settings

   curl -X GET "http://localhost:9200/_cluster/settings?include_defaults=true&pretty"
   

   Confirm cluster.routing.allocation.cluster_concurrent_recoveries, cluster.routing.allocation.node_initial_primaries_recoveries, and any custom cluster.routing.allocation.filter.*.

6. Inspect Index Settings for Shard Count

   curl -X GET "http://localhost:9200/my_index/_settings?pretty"
   

   Check index.number_of_shards and index.number_of_replicas.

7. Monitor Allocation Delay

   curl -X GET "http://localhost:9200/_cluster/health?level=shards&pretty"
   

   Note active_primary_shards vs. active_shards to spot pending allocations.

Fixes Tied to Findings

• Primary Shards > Data Nodes
  • Scale out: add at least one more data node.
  • Reindex with fewer primary shards if adding nodes is not feasible.
• Node Offline / Removed
  • Restart the node or bring it back online.
  • If the node is permanently removed, the cluster will rebalance automatically once enough nodes remain.
• Disk Space Constraints
  • Free space on the affected node(s); move data to another node if possible.
  • Temporarily lower the cluster.routing.allocation.disk.watermark.low and high thresholds to allow allocation, but monitor closely.
• Allocation Filtering or Routing Rules
  • Review cluster.routing.allocation.filter.* settings using the settings API.
  • Remove or adjust filters that prevent shard placement.
  • Example: cluster.routing.allocation.exclude._name: node-01 removes node‑01 from allocation.
• High Allocation Delay
  • Increase cluster.routing.allocation.cluster_concurrent_recoveries to allow more shards to recover concurrently.
  • Check hardware performance (disk I/O, network latency) on nodes with slow recovery.
• Replica Settings
  • Adding replicas can spread data across more nodes, reducing unassigned shards when a node fails.
  • Beware: each replica adds memory overhead and can slow indexing throughput.
  • Adjust via PUT /my_index/_settings:

    curl -X PUT "http://localhost:9200/my_index/_settings" \
     -H "Content-Type: application/json" \
     -d '{"index":{"number_of_replicas":2}}'
    

Escalation Criteria

1. After 30 minutes of unassigned shards, cluster status remains red.
2. Allocation explain consistently shows disk_watermark or node_left without remediation.
3. Cluster health does not improve after adding nodes or freeing disk space.
4. Monitoring alerts (e.g., DataDog, Prometheus) trigger on cluster.health or allocation_delay metrics.

Practical Verification After Fixes

Re‑run the health API:

curl -X GET "http://localhost:9200/_cluster/health?pretty"

Expect status: green and unassigned_shards: 0. If the status is still yellow, double‑check the unassigned_shard list and run _cluster/allocation/explain again to confirm no hidden constraints.

Limitations & Caveats

• Changing the number of primary shards requires reindexing; plan for downtime or use index templates for future indices.
• Increasing replicas improves resilience but consumes more disk and RAM; balance against write latency.
• Allocation filtering is powerful but can inadvertently lock shards into a single node; document any custom filters.
• Disk watermark thresholds are cluster‑wide; avoid setting them too low to prevent frequent rebalancing.

Conclusion

Unassigned shards are the most common cause of Yellow or Red cluster health. By systematically checking shard counts, node availability, disk space, and allocation rules, you can pinpoint the root cause and apply a targeted fix. Remember to verify the cluster returns to Green before concluding the investigation, and monitor key metrics to catch any recurrence early.