Diagnosing Vert.x Event Bus Cluster Failures: A Step-by-Step Guide for Hazelcast 5.x

The Problem: Vert.x Cluster Won't Form or Messages Disappear

If your Vert.x application starts but the Event Bus cluster never forms, or messages sent to remote nodes vanish silently, you're dealing with a common but tricky issue. The symptoms are subtle: logs show Joining cluster... or No members discovered without a clear error, local Event Bus works fine, but remote messages drop. This guide walks through recognizing the condition, diagnosing the root cause, and applying targeted fixes.

Recognizable Conditions

• No cluster formation: Vert.x instances start but never discover each other. Logs contain Joining cluster... followed by No members discovered with no ERROR.
• Local EventBus works, remote fails: Events sent within a single node succeed, but messages sent to other nodes are lost or never arrive.
• Intermittent or duplicate messages: During network partitions or cluster merges, messages arrive out of order or multiple times.

Quick Diagnostic Table

Symptom	Potential Cause	Check
No members discovered	Discovery mismatch (multicast vs TCP-IP)	Match hazelcast.xml discovery settings
Cluster splits unexpectedly	Firewall blocking port 5701	Test with nc -zv <peer> 5701
No message delivery	Address or codec mismatch	Verify EventBus address and codec registration
High latency	GC pauses or thread starvation	Monitor hazelcast.operation.thread.count
Duplicate messages	Split-brain merge conflict	Enable split-brain-protection with minimum-cluster-size

Ordered Diagnostic Checks

1. Version Alignment: Confirm you're using Vert.x 4.5.x with Hazelcast 5.3.x. Mixing Hazelcast 4.x jars causes NoSuchMethodError. Run:

   mvn dependency:tree | grep hazelcast

   All references should resolve to 5.3.x.
2. Network Discovery: Validate hazelcast.xml. In cloud environments, disable multicast and list TCP-IP members explicitly:

   <network>
     <multicast enabled="false"/>
     <tcp-ip enabled="true">
       <member-list>
         <member>10.0.0.1:5701</member>
         <member>10.0.0.2:5701</member>
       </member-list>
     </tcp-ip>
   </network>

   In Kubernetes, use K8s DNS or headless services instead of static IPs.
3. Port Accessibility: Ensure port 5701 is open between nodes. Test with:

   nc -zv 10.0.0.2 5701

   Also verify that bind address and interface regex in config match your network setup.
4. Cluster Size Verification: Use JMX or jconsole to check the Members attribute under com.hazelcast:instance. A healthy cluster should report the expected number of members.
5. EventBus Codec Registration: If using custom codecs, ensure they're registered on both sender and receiver. For POJOs, deploy a shared JAR and register:

   eventBus.registerDefaultCodec(new PojoCodec());

Applying Targeted Fixes

• Discovery Configuration: In cloud or containerized environments, avoid multicast. Use TCP-IP with explicit member lists or Kubernetes discovery plugins. For example, in Kubernetes with dnsPolicy: ClusterFirstWithHostNetwork, use headless service DNS names like my-service-headless.namespace.svc.cluster.local.
• Port and Network Tuning: Open ports 5701–5703 and set auto-increment and port-count in hazelcast.xml to allow multiple instances per host. If using firewall rules, ensure they allow traffic on these ports bidirectionally.
• Codec and Address Consistency: Ensure EventBus addresses match across nodes. If using DeliveryOptions.setCodecName(), the named codec must be registered on the sender. For publish, all consumers must have the codec registered.
• Thread and GC Optimization: Increase hazelcast.operation.thread.count if monitoring shows thread starvation. Tune JVM garbage collection to reduce pause times that can delay message processing.
• Split-Brain Protection: Enable split-brain protection in hazelcast.xml with a minimum-cluster-size greater than half your nodes. This prevents divergent clusters from merging and causing data loss.

When to Escalate

• Cluster fails after 5 minutes with verified config: Engage your network team to investigate firewall rules, routing, or MTU issues.
• Recurrent split-brain with data loss: Consider Hazelcast Enterprise features like WAN replication or external coordination stores like Consul or etcd.
• Persistent message loss despite correct config: Capture a heap dump and thread dumps from affected nodes for deeper analysis.
• CPU >80% consistently on Hazelcast threads: Profile with async-profiler to identify bottlenecks or contention.

Verification and Validation

• Start Vert.x with -Dhazelcast.config=/path/to/hazelcast.xml and monitor logs for successful member discovery. Use jconsole or JMX to confirm cluster size via the Members attribute.
• Deploy a 3-node test cluster in a local environment like kind or k3d. Verify logs show Members [3] and EventBus latency remains below 5ms.
• Simulate a network partition using iptables DROP on port 5701. Observe whether split-brain protection activates and the cluster heals correctly.
• Register a custom codec and send a test message. Verify the consumer receives the correct object type without deserialization errors.

Key Cautions

• Do not mix Hazelcast versions: Vert.x 4.x bundles Hazelcast 5.x. Adding Hazelcast 4.x JARs causes runtime errors.
• Multicast limitations: Hazelcast multicast discovery does not work across VPC peering or Kubernetes services. Use TCP-IP or cloud-specific discovery plugins.
• Codec placement: EventBus send with setCodecName requires the codec on the sender; publish requires it on all consumers.
• Map merge policies: Default merge policies may drop updates during cluster merge. Review for idempotent payloads or customize the merge policy.