Using Moleculer’s Built‑In Circuit Breaker to Guard Service‑to‑Service Calls

Problem: Cascading Failures in a Moleculer Micro‑service Mesh

When one service in a Moleculer‑based mesh begins to return errors or latency spikes, naïve callers keep hammering it, which can exhaust resources, amplify the failure, and eventually bring down unrelated services. This cascading effect is especially dangerous in micro‑service architectures where many services depend on each other.

Thesis: Enable the built‑in circuit breaker middleware to automatically stop calls to a failing service and provide a fallback.

How the Moleculer Circuit Breaker Works

The circuit breaker is a middleware that wraps every action call. It tracks successes and failures within a sliding time window. When the failure ratio exceeds a configurable failureThreshold after at least minRequestCount requests, the breaker opens. While open, further calls are short‑circuited: the middleware either returns a predefined fallbackResponse or throws a MoleculerError. After a halfOpenTime period, a single trial request is allowed; success closes the breaker, failure re‑opens it.

Configuration Example

// broker.js
const { ServiceBroker } = require('moleculer');

const broker = new ServiceBroker({
  nodeID: 'caller-node',
  // global circuit breaker defaults (optional)
  circuitBreaker: {
    enabled: true,
    failureThreshold: 0.5, // 50% failures
    minRequestCount: 5,
    windowTime: 10000, // 10 s
    halfOpenTime: 5000, // 5 s before half‑open
    // fallback response when open
    fallbackResponse: { error: 'Service temporarily unavailable' }
  }
});

// Define a caller service that calls the order service
broker.createService({
  name: 'user',
  actions: {
    async getUserOrder(ctx) {
      // this action is wrapped by the circuit breaker middleware
      return await ctx.call('order.get', { userId: ctx.params.userId });
    }
  }
});

// Order service (simulated)
broker.createService({
  name: 'order',
  actions: {
    async get(ctx) {
      // simulate intermittent failure
      if (Math.random() < 0.8) {
        throw new Error('Order service error');
      }
      return { orderId: 123, amount: 42 };
    }
  }
});

broker.start();

Observing the Breaker State

You can listen to the internal $node.circuitBreaker events or inspect the broker’s metrics.

broker.$node.circuitBreaker.on('open', ({ name, action }) => {
  broker.logger.warn(`Circuit opened for ${name}#${action}`);
});
broker.$node.circuitBreaker.on('halfOpen', ({ name, action }) => {
  broker.logger.info(`Circuit half‑open for ${name}#${action}`);
});
broker.$node.circuitBreaker.on('close', ({ name, action }) => {
  broker.logger.info(`Circuit closed for ${name}#${action}`);
});

Worked Example Walk‑through

1. Start the broker; the caller service begins invoking order.get.
2. Because the order action fails 80 % of the time, after five requests (the minRequestCount) the failure ratio exceeds 0.5, triggering the open state.
3. While open, subsequent calls to order.get are short‑circuited; the caller receives the fallback response ({ error: 'Service temporarily unavailable' }) without invoking the order service.
4. After the halfOpenTime (5 s) elapses, the breaker allows a single trial request. If that request succeeds, the breaker moves to close; otherwise it re‑opens.
5. When the order service is restored (e.g., you lower the artificial error rate), the trial succeeds, the breaker closes, and normal traffic resumes.

Trade‑offs and Limitations

• Setting a low failureThreshold or small windowTime can cause the breaker to open during normal traffic spikes, leading to unnecessary fallback usage.
• The breaker state lives only in the node’s memory; a broker restart loses the state, and in a clustered deployment each node maintains its own breaker. For shared state you would need an external store (e.g., Redis) or a custom middleware.
• The built‑in fallback is static; if you need dynamic fallback logic you must wrap the action yourself or use a plugin.

Actionable Checklist

1. Add the circuitBreaker section to your broker options with values that match your traffic profile (start with the example values and adjust after monitoring).
2. Instrument the $node.circuitBreaker events to log state changes; verify they appear as expected during a failure injection test.
3. After a deployment, watch the logs or metrics for open → halfOpen → close transitions and ensure fallback responses are returned when the circuit is open.
4. If you run multiple broker nodes, consider whether local breakers are sufficient or if you need a centralized store for cross‑node consistency.

By applying the circuit breaker middleware as shown, you protect your Moleculer services from cascading failures while keeping the implementation simple and observable.