Prevent Cascade Failures in Moleculer with Circuit Breaker Middleware

Problem: a slow downstream service drags the whole cluster down

When a Moleculer service calls a dependency that becomes unavailable or slow, every caller waits for timeouts, queues fill, and latency spreads across the system. The useful takeaway is to fail fast instead of waiting. The Circuit Breaker middleware wraps actions and stops invoking them once the failure rate crosses a threshold, returning a CircuitBrokenError immediately.

Desired outcome

Protect a caller action from a failing callee action. The circuit tracks failures in a sliding window, moves CLOSED → OPEN when the error ratio is too high, rejects calls instantly while OPEN, and allows a limited trial in HALF_OPEN to test recovery. Monitoring events let you alert on state changes.

Prerequisites

• A Moleculer broker running in Node.js.
• The circuit breaker middleware package installed in the project root. Run npm i @moleculer/middleware-circuit-breaker as the project owner.
• At least one service with an action that calls a downstream service, or a test action you can make fail.

Install and apply the middleware

Apply globally in moleculer.config.js so all services can opt in via settings.circuitBreaker. Global application does not protect actions unless they declare circuit breaker settings.

// moleculer.config.js
const CircuitBreaker = require("@moleculer/middleware-circuit-breaker");
module.exports = {
  middlewares: [
    CircuitBreaker({
      // Global defaults – can be overridden per service
      threshold: 0.5,          // 50% failure rate
      windowTime: 60000,       // 1 minute sliding window
      minRequestCount: 20,     // Minimum requests before evaluation
      halfOpenRequests: 1     // Trial requests in HALF_OPEN
    })
  ]
};

Configure protection per service or action

Use settings.circuitBreaker on the service. You can also set it on individual actions via action settings.

// services/payment.service.js
module.exports = {
  name: "payment",
  settings: {
    circuitBreaker: {
      threshold: 0.5,
      windowTime: 60000,
      minRequestCount: 20,
      halfOpenRequests: 1
    }
  },
  actions: {
    charge: {
      timeout: 5000,          // Treat slow calls as failures
      circuitBreaker: true,  // Enable breaker for this action
      async handler(ctx) {
        return this.broker.call("gateway.charge", ctx.params);
      }
    }
  }
};

Each option is explained in the settings.circuitBreaker object. The threshold is the failure percentage that triggers OPEN. windowTime is the sliding window in ms for counting requests. minRequestCount prevents opening on a handful of errors. halfOpenRequests is the number of trial calls allowed in HALF_OPEN.

Make failures count correctly

The circuit breaker only counts errors thrown by the action. Slow responses are not failures unless you treat them as such.

• Set an action-level timeout. When the timeout fires, Moleculer throws a RequestTimeoutError which the breaker counts.
• Avoid catching errors inside the action and returning success. Re‑throw or let the broker see the error.
• If the callee is remote, ensure the transporter is shared so retries and errors propagate to the caller.

Observe state changes

The middleware emits circuit.open, circuit.close and circuit.halfOpen events per service instance. Subscribe in the broker startup to log or forward to your observability system.

broker.on("circuit.open", payload => {
  // payload contains service name, action name, stats
  console.warn("Circuit opened", payload);
});

Because state is in‑memory, each service instance evaluates independently. Under uneven load some instances may open while others stay closed, causing partial outages.

Verification steps

1. Create a test service that fails a predictable proportion of the time. Example:

   // services/fail.service.js
   module.exports = {
     name: "fail",
     actions: {
       random: {
         async handler() {
           if (Math.random() < 0.6) throw new Error("Random failure");
           return { ok: true };
         }
       }
     }
   };
   

2. Call the action repeatedly from a script and watch the events. In node test.js:

   const { ServiceBroker } = require("moleculer");
   const broker = new ServiceBroker();
   broker.start().then(async () => {
     for (let i = 0; i < 100; i++) {
       try {
         await broker.call("fail.random");
       } catch (e) {
         // ignore
       }
     }
     broker.stop();
   });
   

3. Verify that the circuit transitions: CLOSED → OPEN when the failure ratio exceeds 50% after at least 20 requests, then HALF_OPEN after windowTime elapses, and finally CLOSED again once trial requests succeed.
4. Simulate a downstream outage by stopping the called service. Verify that the caller immediately receives CircuitBrokenError without blocking the event loop.

Limitations and tuning

• Local state: By default the breaker state is local to each service instance. Under uneven load, some instances may open while others stay closed. For cluster‑wide consistency, use a shared transporter (NATS, Redis) and consider an external state store such as Redis for failure counters.
• Flapping: Setting threshold too low or windowTime too short causes frequent OPEN/CLOSE cycles. Start with defaults and adjust based on observed error rates.
• Latency protection: The breaker does not protect against slow responses. Combine it with the timeout middleware or action-level timeout setting to treat slow calls as failures.
• Automatic retries: CircuitBrokenError is not automatically retried by Moleculer’s internal call mechanism. Configure retryPolicy on the calling action or handle the error at the API Gateway or client.
• Metrics aggregation: Events are emitted per instance. Aggregate them via Moleculer’s built‑in metrics module or an external system like Prometheus to get cluster‑wide visibility.

Recovery and rollback

• If the circuit opens too aggressively, increase threshold, windowTime, or minRequestCount to make it less sensitive.
• To disable protection, remove the circuitBreaker setting from the action or service, or remove the middleware from moleculer.config.js and restart the service. No persistent state is written, so removal reverts to normal calls immediately.