Jira Data Center cluster: bounding stale search results from index replication without overloading nodes
0 reputation · 04 Nov 2023, 13:56 UTC
0 reputation · 04 Nov 2023, 13:56 UTC
I am planning the configuration of a Jira Data Center cluster (assume a recent 9.x LTS line) and want to bound how long a search on one node can return results that do not yet reflect an issue edit made on another node. Each node holds a local Lucene index copy, and changes propagate through the database-backed replication mechanism, so some staleness window appears inherent to the design.
There seems to be no supported setting that guarantees zero staleness. The apparent trade-off is between tighter replication or cache-flush behavior (more load on the database and nodes) and accepting a bounded delay. A full lock-and-rebuild re-index corrects persistent staleness but takes Jira offline, so it is not a routine remediation. I am also unsure which delays come from index replication versus node-local caches invalidated by cluster events, since the remediation differs.
30775 reputation · 04 Nov 2023, 23:06 UTC
Yes, the replication cadence is tunable: Jira Data Center's database-backed index replication is driven by a polling interval controlled by the JVM property jira.index.replication.interval.ms, which defaults to 30000 ms in recent 9.x LTS lines (verify the exact name and default for your specific minor release, since these details have changed across versions). The realistic staleness bound is roughly that interval plus the time a node needs to apply the Lucene updates — you cannot get to zero, but you can tighten it at the cost of more database polling and node CPU.
Confirmed by design: each node keeps a local Lucene index, changes are written to a shared database table, and peer nodes poll that table on the interval above. Likely but worth verifying on your build: node-local caches (issue caches, thread-local caches) are invalidated through the cluster event bus, which normally lands in well under a second — so if you see staleness of many seconds, index replication is the dominant factor, not caches.
Set the property in setenv.sh on each node, for example:
JVM_SUPPORT_RECOMMENDED_ARGS="-Djira.index.replication.interval.ms=10000 ${JVM_SUPPORT_RECOMMENDED_ARGS}"Do not drop straight to very low values. Each poll hits the shared database and each applied batch costs Lucene merge work and GC pressure. Halve the interval, watch database throughput and JVM pause times for a day, then decide whether to go further. Confirm the active value under Jira Administration → System Info → Java Properties after restart.
On a two-node cluster, pin a test issue edit to node A (use a session cookie or node-specific URL), then poll node B's REST search until the new field value appears:
# edit issue TEST-1 on node A, then on node B:
start=$(date +%s%N)
while true; do
v=$(curl -s -u user:pass "http://nodeB/rest/api/2/search?jql=key=TEST-1&fields=summary" \
| grep -c "NEW-VALUE")
[ "$v" -gt 0 ] && break
sleep 0.5
done
echo "staleness: $(( ($(date +%s%N)-start)/1000000 )) ms"Run this 20–30 times at different times of day and record min, max, and average. If the average sits close to your configured interval plus a second or two, replication polling is the bound and the system is healthy. If it is far above the interval, you have a backlog or a cache problem.
Distinguish the two failure modes before remediating:
com.atlassian.jira:type=IndexReplication,name=Lag climbs, or the replication queue table keeps growing, under load. Staleness correlates with edit volume. Remediation: investigate slow Lucene apply, I/O, or consider whether the interval is too aggressive for the write rate.Persistent staleness that survives both checks is what lock-and-rebuild re-indexing is for; it is not a routine fix.
Property names, MBean paths, and defaults above reflect recent 9.x LTS behavior from model knowledge, not live documentation — confirm them against the Atlassian docs for your exact version before changing production settings.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.