Confluence Data Center Clustering: Minimal Viable Architecture and Operational Guardrails

Problem and Takeaway

Running Confluence in a highly available, horizontally scalable configuration requires more than spinning up multiple JVMs. The core challenge is keeping page content, search indexes, and permission checks consistent across nodes without a single point of failure. This article distills the minimal supported architecture for Confluence Data Center, the trust boundaries you must enforce, and the operational checks that prevent silent data divergence.

Requirements Driving the Design

• Active‑active read/write – Every node must serve edits, searches, and permission evaluations with identical results.
• Zero‑downtime rolling upgrades – Nodes can be patched one at a time while the cluster stays online.
• Single‑node failure survival – Loss of any JVM must not lose attachments, index segments, or scheduled jobs.

These requirements rule out simple session replication and force a shared‑storage plus distributed‑cache model.

Smallest Suitable Design

A production‑grade cluster can start with just two nodes. The required infrastructure components are:

• Database – PostgreSQL 13+ or Oracle 19c, single instance (or HA pair) holding all content, permissions, and the cluster lock.
• Shared home – A POSIX‑compliant filesystem (NFSv4 with lockd, SMB3, or GFS2) mounted at $CONFLUENCE_HOME/shared-home on every node. Stores attachments, Lucene index segments, plugin JARs, and the cluster‑safety marker file.
• Hazelcast cache layer – TCP‑IP discovery (multicast does not work in cloud VPCs). Each node runs an embedded Hazelcast member; cache invalidation and cluster‑wide events travel over encrypted TLS channels.
• Load balancer – Sticky sessions are optional; a stateless LB with session affinity disabled works because Confluence stores session data in the database.

Example confluence.cfg.xml snippet for Hazelcast TCP‑IP discovery:

<property name="confluence.cluster.hazelcast.tcpip.enabled">true</property>
<property name="confluence.cluster.hazelcast.tcpip.members">10.0.1.10:5701,10.0.1.11:5701</property>
<property name="confluence.cluster.hazelcast.network.tls.enabled">true</property>

Run this on every node before first startup. Changing discovery mode after bootstrap requires a full cluster restart.

Trust and Data Boundaries

Component	Trust Level	Rationale
Database	Single source of truth	All content, permissions, and the scheduling lock live here. Nodes never trust peer memory for authoritative state.
Shared home	Trusted for blobs and index segments	Attachments and Lucene files are written by the node that owns the operation; other nodes read them. File locking (flock/fcntl) prevents concurrent writes.
Hazelcast	Trusted for cache invalidation only	Encrypted TLS traffic; partition loss triggers rebuild from DB, not peer memory.
Plugin code	Local classpath	JARs are copied to shared home during upgrade, but each node loads its own copy. No distributed classloading.

Operational Checks

Cluster Safety Job

Every 30 seconds (configurable via confluence.cluster.safety.job.interval) each node writes a marker to the CLUSTERSAFETY database table and to cluster-safety.txt on the shared home. If either write fails, the node self‑terminates. Enable debug logging to verify:

log4j.logger.com.atlassian.confluence.cluster.safety=DEBUG

Check the DB table and the file timestamp match within the interval.

Health Endpoint

GET /status returns node state: ACTIVE, PASSIVE, or OFFLINE. Load balancers should remove any node not reporting ACTIVE within the health‑check window.

Index Consistency

On startup each node compares a checksum of the Lucene segments directory against a stored manifest. Mismatch triggers an asynchronous rebuild from the database. Verify with:

curl -u admin:token https://confluence.example.com/rest/api/cluster/index-status

Attachment Integrity

Run the built‑in SHA‑256 manifest check:

https://confluence.example.com/admin/attachment-integrity-check.action

It scans the shared‑home attachment store and reports missing or corrupted files.

Hazelcast Cluster Size

Confirm expected member count via JMX:

jconsole -J-Djava.class.path=/path/to/hazelcast.jar ...
# MBean: com.hazelcast:instance=<node-name>,type=Cluster
# Attribute: Members

Failure Modes and Mitigations

• Split‑brain – Prevented by the database‑backed cluster lock. Only one node can acquire the scheduling lock; others become PASSIVE.
• Lost index segments – Abrupt node kill (kill -9) can leave Lucene segments inconsistent. Always gracefully shutdown (SIGTERM) to flush commits. On restart, the node rebuilds missing segments asynchronously.
• Hazelcast partition loss – Triggers cache rebuild from the database. No user‑visible data loss, but temporary performance dip.
• Shared‑home latency >5 s – Node marks itself unhealthy and is removed from the LB pool. Monitor disk latency (e.g., iostat -x 5) on the NFS/SMB mount.

Conditions That Change the Design

• More than 10 nodes or cross‑region latency >2 ms – Shared‑home Lucene becomes a bottleneck. Replace with a dedicated Elasticsearch remote cluster for search indexing.
• Strict data residency – Deploy separate shared‑home per region with asynchronous database replication (e.g., PostgreSQL logical replication).
• Kubernetes deployment – Replace static shared‑home mount with a CSI volume (e.g., AWS EFS CSI, Azure Disk CSI). Use Hazelcast Kubernetes discovery plugin instead of static TCP‑IP list.

Verification Checklist Before Go‑Live

1. Enable cluster safety logging and confirm marker updates every 30 s in both DB and shared home.
2. Simulate node failure: kill -9 <pid> on one JVM; verify remaining node serves requests within LB health‑check interval and search returns current content.
3. Run attachment integrity check (URL above) – zero mismatches.
4. Validate Hazelcast member count matches node count via JMX.
5. Test rolling upgrade: stage new Confluence version on one node, start, verify ACTIVE, then repeat. No downtime if DB schema compatible.

Limitations and Ongoing Risks

• Shared home must support file locking; NFSv3 without a lock manager causes index corruption. Verify with locktest utility before production.
• Third‑party plugins may not sync automatically via shared home during rolling upgrades – test each marketplace add‑on in a staging cluster.
• Search index on shared home is not transactional; abrupt power loss can still leave segments in an inconsistent state. Use UPS and graceful shutdown scripts.
• Confluence Data Center property names, endpoint paths, and cluster thresholds vary across releases; validate every value in this note against the documentation for your deployed version before production use.

With these guards in place, a two‑node Confluence Data Center cluster delivers the active‑active guarantees Atlassian supports, while keeping operational complexity manageable.