Moleculer Circuit Breaker Tuning: Avoiding False Positives in Node.js Microservices

The Problem: Circuit Breakers Blocking Healthy Services

When implementing circuit breakers in Moleculer, a common pitfall is setting thresholds that trigger false positives during legitimate traffic spikes. This causes healthy services to be temporarily blocked, creating unnecessary downtime and cascading failures in production.

Understanding Moleculer's Circuit Breaker Mechanics

Moleculer's circuit breaker tracks failed requests per service and opens the circuit when failures exceed errorThresholdPercentage. The circuit remains open for resetTimeout milliseconds before allowing a test request through. The key insight is that this percentage is calculated against recent requests, not absolute failure counts.

Minimum Viable Configuration

Start with these conservative defaults that work well for most services:

// In service constructor or broker options
circuitBreaker: {
  // Open circuit after 50% failures
  errorThresholdPercentage: 50,
  // Allow test requests after 10 seconds
  resetTimeout: 10000,
  // Minimum requests before calculating percentage
  minimumCount: 10,
  // Timeout for each individual request
  timeout: 5000
}

Trust and Data Boundaries

Configure circuit breakers at the service level, not globally. Each service should have thresholds tailored to its response time profile and failure modes. Shared configuration creates coupling between unrelated services.

Operational Verification Checklist

1. Simulate failures: Use a test service that throws errors on 50% of requests. Verify the circuit opens after reaching the threshold.
2. Monitor metrics: Track circuitBreaker.opened and circuitBreaker.closed events. A healthy service should rarely see the circuit open.
3. Test load spikes: Generate legitimate traffic that exceeds normal volume but shouldn't trigger failures. Verify the circuit stays closed.
4. Check reset behavior: After the circuit opens, confirm it allows a test request after the configured timeout.

Failure Modes and Detection

Monitor for these warning signs:

• Frequent circuit openings: Indicates thresholds are too aggressive or the service genuinely has reliability issues.
• Long reset times: If resetTimeout is too long, recovery is delayed during actual outages.
• Low minimumCount: Setting this too low causes percentage-based decisions on insufficient data.

When This Design Changes

Reconsider thresholds if:

• Response times consistently exceed 95th percentile of the configured timeout
• Failure patterns shift from transient errors to persistent issues
• Traffic volume drops below the minimumCount threshold regularly

Concrete Example: API Gateway Service

For an API gateway calling multiple backend services, use aggressive timeouts (2-3 seconds) but moderate failure thresholds (70%). This prevents slow backends from blocking the entire gateway while still catching consistently failing services:

const apiService = {
  name: 'api',
  circuitBreaker: {
    errorThresholdPercentage: 70,
    resetTimeout: 5000,
    timeout: 3000,
    minimumCount: 5
  }
}

Verification Command

Run this in a Moleculer service to check current circuit breaker state:

// Check circuit breaker status for a service
const service = broker.services.get('user-service');  // Replace with actual service
if (service) {
  console.log('Circuit breaker:', service.circuitBreaker ? 'enabled' : 'disabled');
  console.log('State:', service.circuitBreaker?.state || 'N/A');
}

Limitations

Moleculer's circuit breaker tracks failures per service instance, not across the entire cluster. Network partitions can cause individual instances to open circuits while others remain available. Additionally, the breaker doesn't distinguish between error types—timeouts and 500 errors are treated equally.

Practical Verification Steps

1. Deploy a test service with the configuration above
2. Make 10 successful requests, then 5 failing requests
3. Verify the circuit opens after the 5th failure (50% threshold)
4. Wait 10 seconds and make another request to confirm the circuit resets