Moleculer's Built-In Circuit Breaker: What It Actually Does and When to Tune It

Your orders service calls payments.charge on every checkout. One afternoon the payments node starts timing out — not crashing, just hanging. Every caller now waits the full action timeout before failing, worker threads pile up, and the whole broker slows down. Moleculer ships with a circuit breaker to stop exactly this cascade, and it's enabled by default. The catch: the defaults are generic, and if you don't understand the failure semantics you can end up with a breaker that trips on validation errors or one that never trips because your retry policy masks the failures.

How the breaker decides to open

The breaker lives in the broker's call path, not in the transporter. That means it behaves identically whether you run NATS, Redis, Kafka, TCP, or the local fake transporter for tests. For each action call, the broker tracks outcomes inside a rolling time window (windowTime, in seconds). When at least minRequestCount calls have occurred in that window and the fraction of failures exceeds threshold (a value between 0 and 1, e.g. 0.5), the circuit opens.

Once open, calls fail fast — they reject immediately instead of burning a timeout waiting on a sick downstream. After halfOpenTime, the breaker lets a single probe request through. If the probe succeeds, the circuit closes and traffic resumes; if it fails, the circuit reopens and the cycle repeats. This gives an overloaded service breathing room without any operator intervention.

One important scoping detail: breaker state is per broker, not per cluster. Each node opens and closes its own circuit independently. During partial degradation you can see inconsistent behavior across nodes — one caller failing fast while another still waits on timeouts. Don't design around the assumption of a globally coordinated breaker.

Configuring it deliberately

The breaker is configured at the broker level. Option names and defaults have shifted across Moleculer releases, so confirm the exact keys against the version pinned in your project before copying this into production:

// moleculer.config.js — verify keys against your installed version
module.exports = {
  circuitBreaker: {
    enabled: true,
    threshold: 0.5,      // open when >50% of calls fail
    windowTime: 60,      // measured over a 60s rolling window
    minRequestCount: 20, // need at least 20 calls before judging
    halfOpenTime: 10 * 1000, // probe after 10s
    check(err) {
      // Only count real infrastructure failures, not business errors
      return err && err.code >= 500;
    }
  }
};

The check function is the most underused knob. By default, any rejected promise counts as a failure. If your actions reject with validation errors (ValidationError, 4xx-style codes) as part of normal business logic, a buggy client sending bad requests can trip the breaker and take the action offline for everyone. A custom check(err) that only counts 5xx-class or timeout errors keeps the breaker focused on actual downstream health.

The retry interaction nobody warns you about

Moleculer's retryPolicy retries a failed call before the breaker observes the final outcome. If you configure 3 retries with backoff plus a 5s timeout, a single logical call can take 20+ seconds and hit the struggling service four times before the breaker counts one failure. Aggressive retries therefore delay breaker tripping and amplify load on the very service the breaker is meant to protect.

A workable pairing: keep retries low (1–2) for calls to flaky downstreams, keep timeouts honest, and let the breaker handle sustained degradation rather than papering over it with retries. Retries are for transient blips; the breaker is for real trouble.

Verifying it works before you need it

You can observe the full state machine locally in a few minutes. Create two services, make the callee throw, and call it in a loop while listening for breaker events:

broker.createService({
  name: 'flaky',
  actions: {
    go() { throw new Error('downstream is sick'); }
  }
});

broker.localBus.on('circuit-breaker.open', () => console.log('OPEN'));
// call broker.call('flaky.go') in a loop and watch for the transition

Run this with the Node.js broker directly (no special permissions needed; it's just a local process). Expected check: after enough failures within the window, calls start rejecting immediately instead of after the timeout — you can confirm the fail-fast behavior by comparing call latency before and after the open event. Then test your custom check by throwing a validation error and confirming the breaker stays closed.

The trade-off: fail-fast looks like an outage

When the circuit is open, callers get immediate rejections. If your API layer doesn't map that to a graceful response — a cached value, a default, a 503 with a retry hint — users see hard errors even though the breaker is working exactly as designed. The breaker buys the downstream time to recover; it does nothing for the caller's experience unless you add fallback handling on top.

Actionable closing

Three things to do this week: (1) pin down the breaker option names for your Moleculer version and set threshold, windowTime, and minRequestCount explicitly rather than inheriting defaults; (2) write a check function that ignores business-logic rejections; (3) audit any action with both retries and the breaker enabled, and lower the retry count so the breaker can actually see failures. Then run the two-service loop test above and watch the open/half-open/close transitions yourself — it's the fastest way to build intuition for how your system will behave on a bad day.