Jira Data Center Index Replication Latency During High-Concurrency Writes

Answer to the Questions

1. Upper bound for replication latency at >500 updates/s
   There is no published hard limit. In practice, observed lag ranges from a few seconds up to several minutes, depending on cluster size, hardware, and network conditions. The replication protocol is fully asynchronous, so the primary node will continue to accept writes while replicas catch up.
2. Back‑pressure or pause on the primary?
   The replication mechanism does not implement back‑pressure. The primary writer keeps receiving updates; replication proceeds in the background. High write rates can increase memory pressure on the primary, so ensure heap sizing is adequate.
3. Configurable thresholds for fail‑over?
   There is no public property such as index.replication.max.lag. Lag is managed internally, and a node that consistently lags may be taken out of the cluster, but the threshold is not exposed to administrators.

Confirmed Facts (Based on Atlassian Documentation & Community Reports)

• Replication is asynchronous and does not pause the primary writer.
• No official configuration setting exposes a replication‑lag threshold.
• Typical lag observed in production ranges from seconds to minutes under sustained high throughput.
• Jira exposes internal replication metrics via JMX beans (e.g., com.atlassian.jira:type=IndexReplication) and the REST endpoint /index/replication.

Practical Steps to Verify and Mitigate Lag

1. Enable and monitor replication metrics:
   • Use JMX to read the ReplicationLag attribute before, during, and after a controlled bulk update.
   • Alternatively, query https://node/rest/api/2/index/replication for live lag data.
2. Simulate a bulk update:
   • Run a script that updates >500 issues per second on a test cluster.
   • Capture logs for replication lag timestamps and verify that the primary continues to accept writes.
3. Assess JVM heap usage:
   • Monitor java.lang:type=Memory MBeans to ensure the primary node does not hit OutOfMemoryError or high GC pause times.
   • Adjust -Xms and -Xmx if necessary.
4. Plan for manual fail‑over:
   • Since the internal threshold is not exposed, schedule periodic health checks on replica nodes.
   • If a node lags beyond a tolerable window (e.g., >5 minutes), consider removing it from the cluster and re‑synchronizing.

Missing Diagnostic Detail Needed for Tailored Guidance

Could you specify the number of nodes in your cluster and the average write latency of your network/storage subsystem? This information would help refine the expected replication lag and recommend precise JVM heap sizing.