Bitbucket Data Center Node Drift Diagnostic Guide

Recognizable condition

In a Bitbucket Data Center cluster behind a load balancer, users sometimes see different results for the same repository depending on which node serves the request. Examples include a repository appearing missing on one node, differing commit counts or pull‑request approval states, and sporadic 500 or 404 errors that resolve on a retry. Load‑balancer health checks remain green because each node answers the probe, but the application view is not consistent across nodes. The drift often appears after a node restart, network partition, database failover, or prolonged NFS latency.

The useful takeaway is to confirm that the problem follows a specific node before touching caches or restarting services. If the symptom moves with the node, the issue is cluster coherence rather than a corrupted repository.

Cause‑to‑symptom table

Observed symptom	Likely cause	Quick check
Same repo shows different commit count or PR approvals per node	Cache incoherence from Hazelcast partition or local cache staleness	Administration > System > Nodes shows mismatched cluster UUID or a node marked DOWN
Repository missing on one node, present on others	Shared home mount inconsistency or NFS latency / stale file handles	File modification timestamps differ across nodes for shared/bitbucket-home
Sporadic 500 on API/UI, resolves on retry	Database replication lag or connection errors causing stale reads	atlassian-bitbucket.log shows database connection errors or unusually long query times
Stuck indexing, outdated search results	Background tasks or reindex jobs stuck on one node	Administration > System > Background Tasks shows running or failed jobs

Hazelcast is the in‑memory data grid Bitbucket Data Center uses for clustering and cache coordination. The shared home directory (typically an NFS mount) holds repositories and metadata. Both must be consistent for a coherent cluster view.

Ordered checks

1. Prove node‑specific drift. Bypass the load balancer and query each node directly using its hostname. Use admin credentials safely—for example, with the -u flag that prompts for a password or by loading credentials from a .netrc file.

   curl -u <admin-user> https://<node-hostname>/rest/api/1.0/projects/<project-key>/repos
   

   Compare the repository list and pull‑request counts across nodes. Record any differences. This step requires Bitbucket admin privileges and network access to the node hostnames.

2. Check cluster health. In the Bitbucket UI, navigate to Administration > System > Nodes (exact path may vary by version; consult your version’s documentation). Verify that all nodes report UP, share the same cluster ID, and run matching versions. Note any node showing DOWN, UNKNOWN, or a different cluster UUID.

3. Inspect logs for partition events. On each node host, as the bitbucket user, open atlassian-bitbucket.log. Look for Hazelcast partition warnings, cache‑miss messages, and database connection errors. Do not restart nodes while reviewing logs.

4. Verify shared home consistency. Confirm that the shared home mount is present and POSIX‑compliant on every node. Compare a recent file’s modification timestamp—for example, a repository metadata file under shared/bitbucket-home—across nodes. Divergent timestamps or different mount options can indicate NFS latency or misconfiguration.

5. Review background tasks. In Administration > System > Background Tasks, check for stuck indexing or replication jobs. Long‑running or failed tasks can leave metadata partially updated.

Fixes tied to findings

Node shows DOWN or different cluster ID. Verify network connectivity between nodes and to the shared database. Perform a rolling restart of the affected node only (never all nodes at once) to avoid quorum loss. Wait for the node to rejoin and show UP with matching cluster UUID before proceeding.

Hazelcast partition or cache incoherence detected. Clear local caches via Administration > System > Caches (path may differ by version). This temporarily increases database load; schedule during low‑usage periods. Allow the cluster to rejoin and propagate cache updates. Avoid simultaneous restarts of multiple nodes.

Shared home NFS latency or mount mismatch. Remount the shared home with identical options across nodes and verify file ownership. Ensure the file system is POSIX compliant. Changes to mounts require service interruption and should be performed during a maintenance window.

Stuck indexing or background tasks. From the admin UI, trigger a reindex for the affected repositories and monitor Background Tasks until completion. Do not force‑kill tasks without guidance from Atlassian Support.

Escalation criteria

Escalate to Atlassian Support when:

• Multiple nodes remain out of sync after a cache clear and a controlled rolling restart.
• Database replication lag persists beyond normal thresholds or logs show repeated connection failures.
• Data loss or repository corruption is suspected.
• Logs show repeated Hazelcast split‑brain events with no network recovery.

Provide node logs, the cluster topology view from Administration > System > Nodes, and relevant database metrics. Avoid changing clustering settings outside a maintenance window.

Verification and limitations

After remediation, repeat the direct‑node check (step 1) and confirm that repository lists and pull‑request counts match across all nodes. Verify that Administration > System > Nodes shows UP status and a identical cluster UUID for every member. Review atlassian-bitbucket.log for the absence of Hazelcast partition warnings and for successful cache‑invalidation entries. Monitor Background Tasks to ensure indexing and replication jobs finish without errors.

Limitations: UI navigation, log file names, and cache‑management options can differ between major Bitbucket Data Center releases. This guide assumes a shared database, shared home directory, Hazelcast‑based clustering, and a load balancer in front of the cluster. Data Center licensing is required for clustering features. Cache clearing will temporarily increase database load.