Diagnosing and Fixing Index Replication Lag in Jira Data Center

Recognizable Symptoms

When the search engine or dashboards show stale data, the following signs often point to an index replication problem:

• Search results or JQL queries omit recently updated issues.
• Dashboard gadgets display counts that lag behind the actual issue count.
• Logs contain IndexReplicationQueue backlog warnings or Lucene index corruption stack traces.
• HTTP 500 errors from the search service rise above 1 % for more than 10 minutes.

Root‑Cause Mapping

Cause	Typical Symptom	Diagnostic Indicator
Network partition between nodes	Stale search across the cluster	Replication queue grows on all nodes
Shared‑home snapshot stall	Missing recent updates	SnapshotManager logs show timeout or retry loops
JVM GC pause >5 s	Search latency spikes	GC logs show long pauses around snapshot times
Disk I/O saturation on shared home	Replication queue backlog	High I/O wait in iostat or vmstat
Version mismatch between nodes	Index corruption on one or more nodes	Logs show incompatible index format errors
Manual rebuild without coordination	Split‑brain state, inconsistent search	Different replicationIndexVersion per node

Verification Checklist

1. Replication status on each node
   Run on every node (replace <JIRA_URL> and <API_TOKEN>):

   curl -u <API_TOKEN> -X GET <JIRA_URL>/rest/api/2/cluster/index/replication/status
   

   Check that replicationIndexVersion matches across nodes and queueSize is zero.
2. Log inspection
   Search atlassian-jira.log for IndexReplicationManager and SnapshotManager entries. Correlate timestamps with GC logs (e.g., gc.log or jvm.log).
3. Shared‑home health
   Verify mount options and usage:

   mount | grep shared-home
   df -h /var/atlassian/application-data/jira/shared-home
   

   Ensure the file system is block‑based (EBS, Persistent Disk) and not NFS with burst latency.
4. Version consistency
   Confirm all nodes run the same Jira major/minor version and Java runtime. On each node:

   ./bin/atlassian-jira.sh -v
   java -version
   

5. Integrity check
   Optionally run a full integrity scan (may be resource intensive):

   curl -u <API_TOKEN> -X POST <JIRA_URL>/rest/api/2/cluster/index/replication/integrity
   

   Expect OK on all nodes.

Common Fixes

• Restore network connectivity
  Verify that all nodes can reach each other on the replication port (default 8080). Use ping or nc -zv from each node.
• Force a snapshot
  If the queue is stuck, trigger a manual snapshot (available on JDC 8.20+):

  curl -u <API_TOKEN> -X POST <JIRA_URL>/rest/api/2/cluster/index/replication/snapshot
  

  Monitor logs for successful snapshot completion.
• Move snapshots to local SSD
  Set jira.index.replication.local=true in jira-application.properties on each node, then restart. This keeps snapshots on local storage, reducing shared‑home I/O.
• Tune GC and heap
  Increase heap size or switch to G1GC if pauses exceed 5 s. Apply changes in jira-application.properties and perform a rolling restart.
• Provision dedicated IOPS for shared home
  If iostat shows high wait times, allocate an SSD with sufficient IOPS or move the shared home to a dedicated block device.
• Coordinated rebuild
  When rebuilding the index, pause all follower nodes, rebuild on the leader, then restart followers one by one. Avoid deleting $SHARED_HOME/index without coordination.

Escalation Thresholds

If after 30 minutes the replication queue remains above 500 k ops, or if integrity checks report CORRUPT on more than one node, elevate the issue to Atlassian support. Also raise an alert if search‑related HTTP 500 errors persist above 1 % for more than 10 minutes.

Post‑Fix Validation

1. Re‑run the replication status API; all nodes should report identical replicationIndexVersion and queueSize=0.
2. Execute a representative JQL query (e.g., updated >= -1h) on each node via REST and compare result counts.
3. Monitor atlassian-jira.log for 15 minutes; no new IndexReplicationQueue WARN entries should appear.
4. Run the integrity API again; all nodes should return OK.
5. Verify dashboard gadgets update within 30 seconds of an issue transition.

Limitations and Practical Checks

• Manual snapshot API is only available on Jira Data Center 8.20+. Earlier releases require a node restart to force a snapshot.
• Enabling local index replication increases per‑node disk usage by roughly the size of the index; plan capacity accordingly.
• Deleting $SHARED_HOME/index forces a full rebuild and can take hours on large installations (≈ 1 M issues per hour on SSD). Schedule during low‑traffic windows.
• GC tuning changes require a rolling restart and should be tested in a staging environment that mirrors production data volume.
• Integrity scans consume CPU and memory; run them during off‑peak periods.

By following this ordered diagnostic workflow, you can isolate the root cause of index replication lag, apply the appropriate remediation, and verify that the cluster returns to a healthy state before resuming normal operations.