Moleculer's Circuit Breaker: Fail Fast on Remote Calls Without the Guesswork

The problem: cascading timeouts on flaky dependencies

You call a remote payments.charge action. The network hiccups. The call hangs for 30 seconds, ties up a thread, and your API gateway returns 504 to the user. Multiply that by a traffic spike and the whole cluster stalls. A circuit breaker stops the bleeding by rejecting calls immediately once the failure rate crosses a threshold—no waiting for timeouts.

Moleculer bakes this into the action definition via the circuitBreaker option. It tracks failures per action, transitions through closed → open → half-open states, and throws a CircuitBreakerError while open so callers can fail fast or trigger a fallback.

What the breaker actually tracks

Only thrown errors and transporter-level failures increment the failure counter. A resolved promise that carries an error payload (e.g., { success: false, error: '...' }) does not trip the breaker unless you explicitly throw it. The breaker is also local to each service node by default—Node A opening its circuit doesn't protect Node B. If you need cluster-wide coordination, you'll need a shared store or an external sidecar.

Key tuning knobs and what they mean

Option	Purpose	Typical starting point
threshold	Failure percentage (0–100) that opens the circuit	50
minRequest	Minimum calls in the window before evaluation starts	10
maxRequest	Max calls tracked in the sliding window	100
timeWindow	Rolling window in milliseconds	60000
resetTime	How long (ms) the circuit stays open before half-open	30000
halfOpenRequests	Probe calls allowed in half-open state	3

Set threshold too low or timeWindow too short and you'll trip on transient blips. Set them too high and the breaker never opens. Start conservative, then tighten after you observe real error distributions.

Worked example: protecting a flaky payment action

// services/payments.service.js
module.exports = {
  name: 'payments',
  actions: {
    charge: {
      // Simulate a flaky downstream: 30% random failure
      async handler(ctx) {
        if (Math.random() < 0.3) {
          throw new Error('Gateway timeout');
        }
        return { transactionId: `txn_${Date.now()}` };
      },
      // Circuit breaker config
      circuitBreaker: {
        enabled: true,
        threshold: 50,        // open at 50% failure rate
        minRequest: 5,        // evaluate after 5 calls
        maxRequest: 20,       // track last 20 calls
        timeWindow: 30000,    // 30-second rolling window
        resetTime: 15000,     // try half-open after 15s
        halfOpenRequests: 2   // allow 2 probe calls
      },
      // Pair with timeout so slow calls don't hang indefinitely
      timeout: 5000
    }
  }
};

Run two brokers locally (NATS or TCP transporter). Call payments.charge in a loop. After ~5 failures out of 10 calls, the breaker opens. Subsequent calls throw CircuitBreakerError instantly—no 5-second timeout wait. Check broker logs for [CIRCUIT-BREAKER] payments.charge OPEN and HALF_OPEN transitions.

Trade-offs and limitations

• Local state only. A failing instance won't protect its peers. For cluster-wide protection, add a shared-state adapter or move breaker logic to the gateway layer.
• Doesn't replace timeouts. The breaker only trips on failures. A slow-but-successful call still burns the full timeout. Always pair circuitBreaker with timeout.
• Resolved error payloads are invisible. If your downstream returns { error: '...' } instead of throwing, the breaker sees success. Normalize to throwing in a wrapper or middleware.
• Tuning is empirical. Defaults work for many services, but bursty traffic or seasonal error patterns need custom windows.

Verify it works in your environment

1. Start two brokers with transporter: 'NATS' (or TCP).
2. Register the payments service above on one broker.
3. From the other broker, call await broker.call('payments.charge', { amount: 100 }) in a tight loop (e.g., 20 calls with 100ms delay).
4. Watch logs: you should see CLOSED → OPEN → HALF_OPEN → CLOSED transitions as the flaky handler fails and recovers.
5. Confirm CircuitBreakerError appears instantly while open (no 5s timeout delay).

Closing: ship with observability, not hope

Moleculer's circuit breaker is a solid first line of defense for remote actions—zero external deps, declarative config, and composable with retry/fallback. But it's only as good as your tuning and visibility. Emit breaker state changes to your metrics stack and alert on OPEN duration. That way you know when a dependency is genuinely unhealthy versus when your thresholds need a tweak.