Question
No handlers error during Cluster Manager membership propagation
Ash Maple
0 reputation · 23 Jan 2021, 21:18 UTC
117.8K views1
Loading…
0 reputation · 23 Jan 2021, 21:18 UTC
In a clustered environment, the Event Bus relies on a Cluster Manager to maintain a shared registry of nodes and their associated addresses. When a message is dispatched to a specific address, the local node utilizes its cached view of the cluster membership to route the request.
There is a known window of inconsistency between the physical failure of a node and the point where the Cluster Manager updates the membership list across all remaining nodes. During this propagation period, the Event Bus may attempt to route messages to a node that is no longer active, potentially leading to delivery timeouts before the system recognizes the absence of handlers.
Given the non-deterministic nature of this update window in distributed setups (e.g., using Hazelcast):
31775 reputation · 24 Jan 2021, 05:27 UTC
The Vert.x Event Bus does not distinguish between a temporary network timeout and a stale membership entry at the routing layer. If a node crashes, the Event Bus will continue to route messages to that node as long as the local cache of the cluster-wide registry still lists it as a handler. This results in a delivery timeout rather than an immediate "no handlers" error.
A NO_HANDLERS failure is triggered immediately only when the following conditions are met at the moment of dispatch:
If a node has crashed but the Cluster Manager has not yet propagated the membership update to the sender, the sender still believes a handler exists. Consequently, the message is dispatched to the dead node, and the system waits for the replyTimeout (default 30 seconds) before failing.
The delay is caused by the failure detection mechanism of the underlying Cluster Manager (e.g., Hazelcast). These managers typically use heartbeats or phi-accrual failure detectors. A node is only marked as dead after a specific number of missed heartbeats. Until that threshold is reached and the membership change is broadcast and processed by the sender, the routing table remains stale.
To reduce the window of inconsistency or handle these failures gracefully, use the following approach:
vertxhazelcast.heartbeat.interval.seconds and hazelcast.max.no.heartbeat.seconds to detect crashes faster (caution: this increases the risk of false positives during GC pauses).DeliveryOptions.setReplyTimeout() to fail faster than the default 30 seconds.Verification Command: To observe this behavior in real-time, enable DEBUG logging for the cluster manager and event bus:
# Example for Logback/Log4j configuration
logger.setLevel("io.vertx.core.eventbus", DEBUG);
logger.setLevel("io.vertx.spi.cluster.hazelcast", DEBUG);Missing Diagnostic Detail: Which Cluster Manager implementation are you using (e.g., Hazelcast, Infinispan, or Consul)? Detection timings vary significantly between these providers.
Use comments to ask for clarification. Post a solution as an answer.
31,775 reputation · 24 Jan 2021, 06:01 UTC
The Vert.x clustered Event Bus maintains a local, eventually consistent view of consumer addresses. When a node crashes, the Cluster Manager (e.g., Hazelcast) eventually evicts it, but until the membership change propagates, the sending node’s cache still lists the dead node as a handler. This matters critically depending on the send mode: send() without a reply handler surfaces no failure at all—the message is simply lost or held until the internal timeout. In contrast, request() triggers a NO_HANDLERS failure only when the local cache reports zero handlers across all known nodes; if a stale entry persists, the sender observes a reply timeout (default 30 s) instead. The distinction is version-sensitive: Vert.x 4.x and 5.x tighten failure detection, but the fundamental eventually consistent routing remains. To verify, start two clustered nodes, register a consumer on one, kill the node, and observe whether the sender gets NO_HANDLERS or a timeout, and how long membership eviction takes.