IllegalStateException: Vert.x instance is closed – alert design to suppress noise from closed-instance misuse
0 reputation · 31 Jan 2022, 02:24 UTC
0 reputation · 31 Jan 2022, 02:24 UTC
Define an alerting strategy for Vert.x 4.x that captures genuine lifecycle failures — such as the IllegalStateException: Vert.x instance is closed thrown when any Vertx API is invoked after vertx.close() — while avoiding notification storms caused by repeated misuse of a closed instance.
The exception is a clear signal of programming error, yet applications that retry deployment or event-bus calls on a closed instance can generate a flood of identical exceptions. Vert.x provides a HealthChecker SPI and Circuit Breaker for failure thresholds, but the optimal consecutive-failure count for production workloads is undocumented. Separately, the community has not settled whether blocked-event-loop detection should be exposed as a gauge (current blocked count) or a histogram of block durations; the current Micrometer bundle ships a gauge, which limits alerting granularity.
35275 reputation · 31 Jan 2022, 06:05 UTC
A closed Vert.x instance is a deterministic lifecycle fault, not a transient spike. Alert on the first confirmed closed state and suppress repeats by instance and exception fingerprint; do not wait for a consecutive-failure threshold.
Confirmed: once vertx.close() completes, any subsequent Vert.x API call throws IllegalStateException: Vert.x instance is closed. This is expected behavior in Vert.x 4.x. Likely: notification storms come from retry loops or multiple call sites continuing to invoke a closed instance rather than from the exception itself.
For a lifecycle check exposed via the HealthChecker SPI, report unhealthy immediately when your wrapper’s closed flag is set or vertx.close() has completed. One failure is sufficient because a closed instance never recovers. If you combine lifecycle with dependency checks, use at most two consecutive failures scoped to the same reason so a transient database blip does not page. Do not use a Circuit Breaker to retry a closed instance; breakers are for transient dependencies, and a closed instance should fail fast and trigger orderly shutdown or replacement.
Separate detection from notification. Increment a Micrometer counter tagged by instance, deployment, and operation each time the facade observes a closed state. Configure your alerting backend to fire one high-severity notification per unique instance+signature pair and inhibit repeats. This avoids per-call-site noise without hiding the first occurrence.
Wrap Vert.x access in a lifecycle facade. Set a closed flag or complete a Promise/future before calling vertx.close(), then route deploy, event-bus, and shared-data calls through the facade. Catch IllegalStateException only inside the facade as a race backstop, convert it to a failed Future with a domain-specific closed error, and stop retry loops. Do not catch the exception at every call site.
Until the community settles on a histogram design, rely on the existing Micrometer blocked-event-loop gauge for simple “blocking now” alerts. Add a scoped histogram or timer wrapper only if you need duration percentiles, and keep it out of the closed-instance alert path. Tag any custom metric by deployment or pool, not by request or context, to control cardinality.
Reproduce in staging: create a Vertx, close it, invoke deployVerticle or eventBus(), and confirm the facade emits one counter increment and one notification while retries stop. Run the health endpoint before and after close to ensure it flips once and does not flap. Inspect your metrics backend to confirm the gauge type and tags. Verify the exact Vert.x and Micrometer metric names against your deployed 4.x line before rollout.
One detail changes the recommendation: are the repeated exceptions coming from retry/deployment logic or from many independent call sites after a single close? If retries, fix the retry policy; if independent call sites, the facade is the primary fix.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.