Diagnosing Missing Entities in Akka Cluster Sharding

You send a message to a sharded entity and nothing happens. No error, no response — the entity simply never materializes, or it existed yesterday and is gone today. This is one of the most common Akka Cluster Sharding complaints, and it almost always traces back to one of three causes: the entity type was never registered on the node holding the shard, the entity passivated and failed to recover, or serialization mismatches silently broke message delivery. This guide walks through the checks in the order that finds the problem fastest.

Version assumptions: examples target Akka 2.6/2.7 with the typed API (akka-cluster-sharding-typed). Entity ID serialization changed around Akka 2.8, so if you are mid-upgrade, treat ID handling as a suspect too and verify against the release notes for your exact version.

The recognizable condition

Typical symptoms:

• Messages sent via EntityRef get no reply and time out at the ask level.
• Logs show Unregistered entity or messages being buffered indefinitely.
• An entity that worked before a rolling restart or rebalancing no longer responds.
• After an idle period, the entity "disappears" and its state is lost on next access.

Cause and diagnostic table

Check 1: Is the entity type registered everywhere it needs to be?

Sharding only starts entities on nodes that have registered the EntityTypeKey. A common mistake is registering on the node that sends messages but not on the nodes that host shards, or gating registration behind a cluster role that some nodes do not have.

// Run on every node that should host entities (typically guarded by a role)
val sharding = ClusterSharding(system)
sharding.init(Entity(CounterEntity.TypeKey) { entityContext =>
  CounterEntity(entityContext.entityId)
})

Verify the node's roles match the sharding role setting in configuration. If akka.cluster.sharding.role = "backend" is set, only nodes with akka.cluster.roles = ["backend"] will host regions. A mismatch here produces exactly the "message sent, nothing happens" symptom. Check startup logs for the region starting on each expected node.

Check 2: Passivation and recovery

Entities passivate after an idle timeout (configurable via passivate-idle-entity-after in recent versions, or via an explicit Passivate message). Passivation itself is normal — the failure mode is that the entity cannot come back. For persistent entities, that means the journal or snapshot store is unreachable or the entity's events fail to deserialize. For non-persistent entities, state is simply gone unless you enabled remember-entities.

Ordered checks:

1. Confirm the journal plugin (akka.persistence.journal.plugin) points at a store that is actually running and reachable from every node.
2. Look for recovery errors in logs at entity restart — deserialization failures here often appear as the entity starting and immediately stopping.
3. If you rely on entities surviving rebalancing or restarts, confirm akka.cluster.sharding.remember-entities = on (and understand its storage cost) or use event sourcing.

Check 3: Serialization and dispatcher consistency

Sharding moves entity envelopes across the wire. If a message class is not bound to a working serializer on every node, delivery fails — sometimes silently at the default logging level. Likewise, if the entity ID extraction differs between nodes (a real hazard across the 2.7→2.8 boundary), two nodes can disagree about which entity a message targets.

1. Bind every message type in application.conf, identically on all nodes, e.g. Jackson or a Protobuf serializer.
2. Temporarily raise logging: akka.actor.debug.unhandled = on and sharding debug logging, then resend the failing message and watch for dropped-message or serialization warnings.
3. If you upgraded Akka recently, confirm entity ID serialization is compatible across old and new nodes during the rolling upgrade.

Fixes tied to findings

• Registration gap: move sharding.init so it runs unconditionally on every hosting node, or fix the role list. Restart affected nodes.
• Recovery failure: restore journal connectivity, then fix or migrate the offending event schema. Entities will recover on next message.
• Serializer mismatch: align bindings and redeploy. There is no state change, so no rollback is needed beyond reverting a bad config.
• Split-brain allocation: enable the split-brain resolver (or lease-based sharding) with a quorum appropriate to your cluster size before re-enabling traffic.

Verifying the fix

Stand up a minimal two-node local cluster (different ports, same seed config), register a trivial entity type, and send one message. You should see log lines like Entity [some-id] started and a reply at the sender. The absence of Unregistered entity warnings during shard rebalancing is the signal that registration and roles are correct. Then kill one node and confirm the entity is recreated on the survivor with its state intact (if persistent).

When to escalate

Escalate to the Akka forums or your support channel when: entities disappear with correct registration, healthy persistence, and consistent serializers; shard allocation flaps between nodes without a network partition; or you hit ID inconsistency only during a 2.7→2.8 rolling upgrade. Capture sharding debug logs, the cluster membership events around the incident, and your application.conf sharding section — those three artifacts answer most follow-up questions.