Diagnosing Cloudflare Workers Durable Objects: Common Production Issues and Fixes

Why Durable Objects Fail in Production

Durable Objects give you strongly consistent state and coordination at the edge, but their programming model introduces failure modes that don't appear in stateless Workers. This guide maps the most frequent production symptoms to root causes, shows the minimal diagnostic steps to confirm each, and provides targeted fixes with escalation paths.

Quick Reference: Symptom → Cause → First Check

Observable Condition	Typical Cause	First Diagnostic Step
Inconsistent state reads for the same object ID	Stale in-memory cache after external storage write	Compare ctx.storage.get() vs ctx.storage.getAll() vs direct SQL query
WebSocket drops silently after ~60s idle	Edge TCP idle timeout; missing hibernation setup	Check upgrade response for cf-durable-object-id and hibernatable: true in acceptWebSocket()
TransactionAbortedError storms under load	Optimistic concurrency conflict on same keys	Add retry with backoff; log e.details to identify conflicting keys
Migration (wrangler deploy --migrate) stalls or fails	In-flight requests to old class version; drain timeout	Check dashboard Workers → Durable Objects → Migration status; run wrangler durable-objects list
Alarms fire late or not at all	Alarm coalescing (only one alarm per object)	Log await ctx.storage.getAlarm() before each setAlarm(); detect overwrites
RPC/Service Binding returns TypeError: Not a DurableObjectStub or hangs	Binding name mismatch or stub used after request context ends	Verify [services] binding name matches env.MY_DO.get(id); ensure stub used within same fetch()
ctx.blockConcurrencyWhile() runs concurrently	Missing await in callback or unguarded entry points	Audit all handlers (fetch, alarm, webSocketMessage); add console.assert(!ctx.isConcurrencyBlocked) in tests

Detailed Diagnostics and Fixes

1. Inconsistent State Reads Across Requests

Condition: Two requests to the same Durable Object ID return different values for the same key, even though no explicit write occurred between them.

Cause: The object instance holds an in-memory cache. If another instance (or a direct SQL API write) modifies the underlying storage, the cache becomes stale until the next get() or getAll() call refreshes it.

Diagnostic:

1. In the object's fetch() handler, log await ctx.storage.get('key'), await ctx.storage.getAll(), and a direct SQL query (if using SQL API) for the same key.
2. Check whether ctx.blockConcurrencyWhile() wraps all storage mutations. Gaps allow concurrent writes that bypass the cache invalidation.

Fix: Ensure every write path (including alarms, WebSocket messages, and RPC calls) uses ctx.blockConcurrencyWhile(async () => { ... }). For reads that must be fresh, call ctx.storage.getAll() or use a transaction with txn.get().

// Correct pattern: wrap all mutations
export class MyObject {
  constructor(ctx, env) { this.ctx = ctx; }
  async fetch(req) {
    return this.ctx.blockConcurrencyWhile(async () => {
      const val = await this.ctx.storage.get('counter');
      await this.ctx.storage.put('counter', (val ?? 0) + 1);
      return new Response(String(val));
    });
  }
}

Verification: Run wrangler dev --persist-to=.wrangler/state and send concurrent requests to the same object ID. Observe logs: each increment should be serialized and visible to the next request.

2. WebSocket Connections Drop After ~1 Minute Idle

Condition: A client WebSocket stays open but stops receiving messages after roughly 60 seconds of inactivity. No close event fires on the client.

Cause: Cloudflare's edge terminates idle TCP connections at ~60s. Durable Objects can hibernate WebSockets, but only if you explicitly opt in with acceptWebSocket(ws, { hibernatable: true }) and implement webSocketMessage and webSocketClose handlers.

Diagnostic:

1. Inspect the WebSocket upgrade response headers (via browser devtools or wscat -v). Look for cf-durable-object-id — its presence confirms the DO accepted the socket.
2. Verify the fetch() handler calls ctx.acceptWebSocket(ws, { hibernatable: true }).
3. Confirm the class exports async webSocketMessage(ws, message) { ... } and async webSocketClose(ws, code, reason) { ... }.

Fix: Update the fetch() handler and add the required handlers:

export class ChatRoom {
  constructor(ctx, env) { this.ctx = ctx; }
  async fetch(req) {
    const upgradeHeader = req.headers.get('Upgrade');
    if (upgradeHeader !== 'websocket') {
      return new Response('Expected WebSocket', { status: 400 });
    }
    const [client, server] = Object.values(new WebSocketPair());
    this.ctx.acceptWebSocket(server, { hibernatable: true });
    return new Response(null, { status: 101, webSocket: client });
  }
  async webSocketMessage(ws, message) {
    // Handle incoming message; ws.send() works while hibernated
    ws.send(`Echo: ${message}`);
  }
  async webSocketClose(ws, code, reason) {
    // Cleanup if needed
  }
}

Risk: Hibernated WebSockets accrue billable duration (durable_objects_hibernated_web_socket_duration metric). Monitor costs if you have many long-lived idle connections.

Verification: Deploy a test Worker, connect with wscat -c wss://your-worker.your-subdomain.workers.dev, wait 70 seconds, then send a message. The webSocketMessage handler should fire and the connection should persist.

3. Repeated TransactionAbortedError Under Load

Condition: ctx.storage.transaction() throws TransactionAbortedError frequently when traffic spikes.

Cause: Optimistic concurrency control: another transaction committed changes to one of the keys your transaction read. The runtime aborts and retries automatically up to a point, but under contention the default retries may exhaust.

Diagnostic: Wrap the transaction in a try/catch and log e.details (contains conflicting key names). Example:

try {
  await this.ctx.storage.transaction(async (txn) => {
    // ... read/write keys
  });
} catch (e) {
  if (e.name === 'TransactionAbortedError') {
    console.log('Conflict on keys:', e.details);
  }
  throw e;
}

Fix: Implement application-level retry with exponential backoff (max 3–5 attempts). For non-critical writes where last-write-wins is acceptable, use put({ key: val, allowUnconfirmed: true }) to bypass the transaction.

async function txnWithRetry(fn, maxAttempts = 5) {
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (e) {
      if (e.name !== 'TransactionAbortedError' || attempt === maxAttempts) throw e;
      const delay = Math.min(50 * 2 ** attempt, 1000) + Math.random() * 50;
      await new Promise(r => setTimeout(r, delay));
    }
  }
}

Verification: Write a test that launches two parallel transactions modifying the same key. Confirm the retry logic succeeds and logs the conflict.

4. Migration Stalls or Fails with DurableObjectNamespaceMigrationError

Condition: wrangler deploy --migrate hangs or exits with a migration error.

Cause: The migration waits for all in-flight requests to the old class version to drain (default 30s). Long-running requests (e.g., WebSockets, streaming responses) can exceed this.

Diagnostic:

1. Open Cloudflare Dashboard → Workers → your Worker → Durable Objects → Migration status.
2. Run wrangler durable-objects list --namespace=<namespace_name> to see object IDs stuck in the old version.

Fix: Increase drainTimeoutSeconds in wrangler.toml (max 300s). For WebSockets, ensure the old class's webSocketClose handler closes connections promptly on migration signal.

# wrangler.toml
[durable_objects.migrations]
  [[durable_objects.migrations]]
    tag = "v2"
    new_classes = ["MyObjectV2"]
    # optional: extend drain timeout
    drain_timeout_seconds = 120

Limitation: Maximum drainTimeoutSeconds is 300s. Longer drains require manual coordination: stop traffic (e.g., via routing rules), wait for in-flight requests to finish, then deploy.

Verification: Change the Durable Object class name, add a migration entry, run wrangler deploy --dry-run then wrangler deploy. Monitor the dashboard migration status until it shows "Completed".

5. Alarms Fire Late or Not at All

Condition: ctx.storage.setAlarm() doesn't trigger at the expected time, or a later alarm overwrites an earlier one.

Cause: Each Durable Object can have only one pending alarm. Calling setAlarm() again before the previous alarm fires replaces it (coalescing). Minimum granularity is ~10 seconds.

Diagnostic: Before every setAlarm(), log await ctx.storage.getAlarm(). Compare the returned timestamp with the new one to detect overwrites.

Fix: Store multiple due timestamps in a sorted set (e.g., a key holding [{due: number, payload: any}]) and process all due items in the single alarm() handler:

async alarm() {
  const now = Date.now();
  const timers = (await this.ctx.storage.get('timers')) ?? [];
  const due = timers.filter(t => t.due <= now);
  const remaining = timers.filter(t => t.due > now);
  await this.ctx.storage.put('timers', remaining);
  for (const t of due) {
    // process t.payload
  }
  if (remaining.length > 0) {
    await this.ctx.storage.setAlarm(remaining[0].due);
  }
}

async function scheduleTimer(dueMs, payload) {
  const timers = (await this.ctx.storage.get('timers')) ?? [];
  timers.push({ due: dueMs, payload });
  timers.sort((a, b) => a.due - b.due);
  await this.ctx.storage.put('timers', timers);
  await this.ctx.storage.setAlarm(timers[0].due);
}

Verification: Set an alarm for 30s in the future, then immediately set another for 15s. Verify only the later alarm fires and the handler processes both due timestamps from the stored sorted set.

6. RPC/Service Binding Errors: TypeError: Not a DurableObjectStub

Condition: Calling env.MY_DO.get(id).fetch() from another Worker throws a type error or hangs indefinitely.

Cause: Two common mistakes: (1) the binding name in wrangler.toml under [services] doesn't match the variable used in code; (2) the stub is captured and used after the originating fetch() handler has returned (stubs are bound to the request context).

Diagnostic:

1. Check wrangler.toml: [services] MY_DO = "my-namespace" must match env.MY_DO in the calling Worker.
2. Ensure stub usage occurs synchronously within the same fetch() handler, or re-create the stub via env.MY_DO.get(id) when needed later.

Fix: Correct the binding name and never store stubs in ctx.storage or pass them via Queues. Re-create on demand:

// Caller Worker
export default {
  async fetch(req, env) {
    const id = env.MY_DO.idFromName('room-1');
    const stub = env.MY_DO.get(id); // use immediately
    const resp = await stub.fetch('https://do/internal-action');
    return resp;
  }
};

Verification: Deploy both Workers, call the caller Worker's endpoint, and confirm the RPC succeeds. Then modify the caller to store the stub in a global variable and use it in a subsequent request — it should fail, confirming the context bound.

7. ctx.blockConcurrencyWhile() Does Not Serialize

Condition: Despite wrapping storage mutations in blockConcurrencyWhile(), you observe concurrent execution (e.g., lost updates, interleaved logs).

Cause: The callback returns a promise that resolves before the async work completes (missing await), or not all entry points (fetch, alarm, webSocketMessage, webSocketClose) are wrapped.

Diagnostic: Add console.assert(!ctx.isConcurrencyBlocked, 'Concurrency already blocked') at the start of each handler during testing. If it fires, you have nested or missing blocks.

Fix: Ensure every public handler that mutates storage uses blockConcurrencyWhile and awaits the returned promise:

async fetch(req) {
  return this.ctx.blockConcurrencyWhile(async () => {
    // all async storage ops here
    await this.ctx.storage.put('key', 'value');
  });
}

async alarm() {
  return this.ctx.blockConcurrencyWhile(async () => {
    // alarm mutation
  });
}

async webSocketMessage(ws, msg) {
  return this.ctx.blockConcurrencyWhile(async () => {
    // mutation from WS
  });
}

Verification: Run wrangler dev --persist-to=.wrangler/state and simulate concurrent fetches, alarms, and WebSocket messages to the same object ID. Log entry/exit of each blockConcurrencyWhile block; they should never overlap.

Escalation Criteria

Engage Cloudflare support (or your account team) when:

• Migration remains stuck after increasing drainTimeoutSeconds to 300s and draining traffic manually.
• TransactionAbortedError persists despite correct retry/backoff and low contention (suggests platform issue).
• WebSocket hibernation fails with correct hibernatable: true and handlers present (check for cf-durable-object-id header).
• Alarms never fire even with a single setAlarm() and no overwrites (verify getAlarm() returns the set timestamp).

Verification Checklist for Staging

1. Concurrency: Run wrangler dev --persist-to=.wrangler/state; hammer the same object ID with 50 parallel requests; assert serial execution via logs.
2. WebSocket hibernation: Deploy test DO; connect with wscat; idle 70s; send message; verify webSocketMessage fires.
3. Transaction retry: Inject two parallel transactions modifying the same key; confirm backoff succeeds.
4. Migration: Change class name, add migration entry, run wrangler deploy --dry-run then wrangler deploy; watch dashboard.
5. Alarm coalescing: Set alarm for 30s, then 15s; verify only later fires and handler processes both from stored sorted set.
6. RPC binding: Call env.MY_DO.get(id).fetch() from another Worker; verify stub works within handler and fails after response ends.

Key Limitations to Remember

• Durable Object storage is per-object — ctx.storage keys are scoped to the object ID. Cross-object queries require the SQL API or an external database.
• Hibernatable WebSockets incur billable duration while hibernated; monitor durable_objects_hibernated_web_socket_duration.
• Migration drainTimeoutSeconds max is 300s; longer drains need manual traffic stop.
• Alarms have minimum ~10s granularity; sub-second scheduling requires an external trigger (e.g., Queue + consumer).
• RPC stubs are not serializable — cannot store in ctx.storage or pass via Queue messages; re-create via env.NAMESPACE.get(id).
• Local wrangler dev uses in-memory storage by default; enable --persist-to for durable local testing.