Moleculer Retry and Fallback Policies Without Duplicate Side Effects

If orders.create calls a downstream service that fails twice and then succeeds, a plain Moleculer call returns an error to the caller even though one more attempt would have worked. The useful takeaway: enable action-level retries only when the downstream operation is idempotent or protected by an idempotency key, attach that key before retries run, and treat fallback as a visible degraded response rather than a silent success.

Desired outcome

You want transient downstream failures to be retried automatically, while deterministic failures such as validation errors or ServiceNotFoundError fail fast. You also want a fallback response only after retries are exhausted, and you want monitoring to still see the original failure.

Assumption: a Moleculer 0.x broker running in a Node service. Moleculer 0.x option names and defaults have shifted across releases, so verify the exact schema against the package installed in your project before deploying.

Prerequisites

1. A Moleculer broker with at least one action that calls another service.
2. A downstream action that is safe to retry, or a downstream API that accepts an idempotency key.
3. A non-production downstream stub or sandbox for verification.
4. Access to broker logs or metrics so retry and fallback behavior can be observed.

Run these checks in the service project root as the normal service user; no elevated permissions are required:

npm ls moleculer
node -p "require('moleculer/package.json').version"

Expected check: one installed Moleculer version is reported. Risk: if multiple copies are installed, middleware and schema behavior may differ between packages.

Configure action-level retries and fallback

Moleculer lets you define retries and fallback directly on an action, or set a broker-wide retryPolicy. The fallback method receives the context and error, and it is invoked only after retries are exhausted, not on the first failure.

Use this shape only after confirming that your installed version supports these fields:

const crypto = require('crypto');

module.exports = {
  name: 'orders',
  actions: {
    create: {
      retries: 3,
      fallback(ctx, err) {
        broker.logger.warn('orders.create fallback', { id: ctx.id, error: err.message });
        return { id: null, status: 'accepted-pending', degraded: true };
      },
      async handler(ctx) {
        return broker.call('DOWNSTREAM_SERVICE.ACTION', {
          amount: ctx.params.amount,
          idempotencyKey: ctx.meta.idempotencyKey
        });
      }
    }
  }
};

The fallback above is intentionally marked as degraded. Do not return a value that looks like a normal success unless callers and dashboards can distinguish it.

Attach an idempotency key before retries run

Retrying non-idempotent actions such as payment or write operations can duplicate side effects unless the downstream service supports idempotency keys. A broker middleware can add a key to ctx.meta before the action handler executes:

const crypto = require('crypto');

broker.use({
  localAction(next, action) {
    return async function wrapped(ctx) {
      if (action.retries && !ctx.meta.idempotencyKey) {
        ctx.meta.idempotencyKey = crypto.randomBytes(16).toString('hex');
      }
      return next(ctx);
    };
  }
});

This middleware changes behavior for every retried action. If your downstream service does not honor the key, retries remain unsafe for writes.

Add backoff outside the action schema

Retry delays and jitter are not built into the core action schema in the versions covered here. Teams typically wrap the handler or use middleware to add exponential backoff between attempts. Use one retry mechanism per call path so attempts are not multiplied.

If your action schema does not provide delay control, wrap the downstream call inside the handler:

const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

async function withBackoff(fn, { retries = 3, base = 100, max = 2000 } = {}) {
  let lastErr;
  for (let attempt = 1; attempt <= retries; attempt++) {
    try {
      return await fn();
    } catch (err) {
      lastErr = err;
      if (attempt === retries) break;
      const delay = Math.min(max, base * 2 ** (attempt - 1));
      await sleep(delay + Math.random() * base);
    }
  }
  throw lastErr;
}

If you also set action-level retries, verify the effective attempt count in staging. Broker-level retryPolicy may override action-level settings depending on version precedence.

Focused procedure

1. Classify the downstream call: read-only, idempotent write, or non-idempotent write.
2. For read-only or key-protected calls, set retries on the action and add a fallback that returns a degraded response.
3. For non-idempotent writes without an idempotency key, leave retries disabled or require the key upstream.
4. Add the idempotency middleware if retries are enabled.
5. Add backoff in the downstream wrapper or middleware, but do not stack two independent retry loops.
6. Deploy to staging and run the verification below before production.

Expected checks

Scenario	Expected behavior	What to inspect
Stub fails first two attempts, then succeeds	Third attempt succeeds and returns the expected payload	Broker logs or metrics for attempt count; downstream stub counter
Downstream disabled	Fallback executes only after retries are exhausted	Fallback log entry; degraded response returned to caller
Validation error	No retry by default	Single downstream invocation; error surfaces to caller
Missing action name	ServiceNotFoundError is not retried by default	No repeated routing attempts in logs
Write action with idempotency key	No unexpected duplicate writes	Downstream audit log or unique-key table

A practical local check is to run a test broker with a stubbed downstream action that fails the first two calls and then succeeds:

node scripts/retry-fallback-check.js

Run this in a local or CI environment, not against production. Expected check: the third attempt returns the payload, and the fallback path is not used. Risk: if the script points at a real downstream service, it may perform real writes.

Recovery options

• If fallback responses are hiding an outage, add a metric or alert on fallback execution and make the degraded flag visible to callers.
• If duplicate writes appear, disable retries for that action, enforce the idempotency key downstream, and reconcile any duplicated records.
• If broker-level retryPolicy overrides your action settings, revert the broker option and retest the effective configuration.
• If you deployed this configuration, rollback by reverting the action schema and broker options; no data rollback is needed unless downstream writes already occurred.

Limitations and version verification

Moleculer does not include a native circuit breaker. For high-failure scenarios, combine retries with external middleware or a service mesh, but only after the basic retry and fallback behavior is verified.

The broker prefers local service instances by default when multiple nodes host the same action, reducing network hops and latency. Confirm whether a retry re-resolves the endpoint in your version; if it does not, retries may hit the same failed instance.

Exact option names and defaults have shifted across Moleculer 0.x releases. Review the Moleculer package source or changelog in your project to confirm the supported retries, fallback, and retryPolicy fields before relying on them in production.