Protecting Microservices with Moleculer Bulkhead: A Hands‑On Guide

Problem: Overloaded Services and Cascading Failures

When a microservice receives more requests than it can handle, the backlog grows, memory spikes, and the whole system can start to fail. In a distributed environment, one overloaded service can bring down dependent services, creating a cascading failure. The question is: how do we isolate a hot spot so that the rest of the system keeps running?

Bulkhead – The Built‑in Isolation Middleware

Moleculer ships a Bulkhead middleware that caps the number of concurrent executions of an action. When the limit is hit, incoming requests are either queued or rejected, optionally triggering a fallback function. This pattern is inspired by the architectural concept of "bulkheads" in ships: compartments that keep the vessel afloat even if one section takes on water.

Configuring Bulkhead

Bulkhead is applied per‑action in the service definition. The schema accepts several options:

• concurrency – maximum simultaneous executions (default 1).
• timeout – maximum time (ms) a request can wait before being rejected.
• queue – maximum queued requests. If omitted, the queue is infinite.
• fallback – a function that returns a response when the action is busy.

Example:

module.exports = {
  name: "payment",
  actions: {
    process: {
      async handler(ctx) {
        // heavy async work
        return await heavyPayment(ctx);
      },
      bulkhead: {
        concurrency: 3,
        timeout: 2000,
        queue: 10,
        fallback: () => ({ error: "Too busy, try later" })
      }
    }
  }
};

Queue vs Immediate Rejection

If queue is set, excess requests sit in a FIFO queue until a slot frees up or timeout expires. Without a queue, the middleware rejects immediately with the fallback response. Choosing between them depends on whether you prefer to let clients wait or to fail fast.

Practical Example: 10 Concurrent Requests

1. Create payment.service.js with the configuration above. Install Moleculer:
npm i moleculer

2. Start the broker:
node -e "require('moleculer').ServiceBroker({ nodeID: 'node1' }).start();"

3. In a separate terminal, send 10 parallel HTTP POSTs to the action (assuming the service is exposed via Moleculer‑REST):
for i in {1..10}; do curl -s -X POST http://localhost:3000/payment.process -d '{}' & done wait

4. Observe the logs. You should see only 3 actions running concurrently. The remaining 7 either queue (if queue is set) or hit the fallback after timeout seconds. Example log snippet:

[Bulkhead] payment.process is busy – queued 4 requests
[Bulkhead] payment.process is busy – fallback invoked

5. Verify the response of a queued request after 2 seconds. It should return the fallback JSON {\"error\":\"Too busy, try later\"} if the queue is exhausted or the timeout is reached.

Trade‑offs & Limitations

• Memory & CPU Overhead: Each action maintains a semaphore and an optional queue, adding small but measurable overhead.
• Latency Increase: Queued requests experience added wait time, which may affect SLAs.
• Queue Overflow: If the queue grows beyond the broker’s internal limits, messages may be dropped.
• Synchronous Actions: Bulkhead only protects async actions. CPU‑bound sync code should be offloaded to worker threads.
• Tuning Required: Setting concurrency too low can starve the service; too high defeats the purpose.

Takeaways & Next Steps

Bulkhead is a lightweight, declarative way to guard your services against overload. Pair it with Moleculer’s Circuit Breaker for full resilience: the circuit breaker trips when the service consistently fails, while the bulkhead limits concurrent load. Start with conservative limits, monitor Bulkhead metrics (e.g., bulkhead.concurrency), and adjust based on real traffic patterns.

Ready to protect your microservices? Add the bulkhead config to your critical actions and observe how the system stays afloat during traffic spikes.