Akka Cluster Sharding: remember-entities vs passivation-only for sub-second recovery SLO

Direct answer

For a single entity type with roughly 120,000 entities on Akka 2.9, the recommended pattern is to enable rememberEntities = true and tune the persistence path to bring replay under 800 ms. Passivation-only recovery depends on external traffic and cannot guarantee a strict p99 SLO during burst rebalances. Splitting into hot and cold entity types is the only hybrid mode Akka core supports, but you do not need two full sharding topologies if the journal replay is fast enough.

Confirmed behavior in Akka 2.9.x

• rememberEntities is an entity-type-level flag; there is no runtime toggle in the 2.9.x line. Changing it requires a rolling restart or a blue/green entity-type migration.
• A custom ShardAllocationStrategy controls shard placement only; it cannot influence whether an entity is restarted.
• Akka Distributed Data tracks cluster membership and shard state, not per-entity liveness. Using it as a hot-entity registry would require manual StartEntity messages and adds operational risk.
• There is no built-in TTL-backed hybrid mode in Akka core.

Likely explanation for the 1.2–3 s replay latency

The observed latency most likely comes from cross-datacenter Cassandra reads or missing snapshots rather than inherent Akka overhead. With ~5 KB of state per entity, snapshotting every 50–100 events typically reduces replay to a few hundred milliseconds when reads stay in the local datacenter.

Tuning steps for sub-800 ms recovery

1. Keep Cassandra reads local. Set read-consistency = LOCAL_ONE (or LOCAL_QUORUM if stale entity IDs are unacceptable) and ensure journal and snapshot tables use NetworkTopologyStrategy with local-DC replication only.
2. Snapshot aggressively. Lower snapshot-after to 50–100 events and set snapshot-isolation = off for the Cassandra plugin to avoid lightweight-transaction overhead. Provision Cassandra write capacity for 2–3× the current event rate.
3. Increase shard count. Move to at least 600 shards (10× your node count). This caps entities-per-shard near 200 and keeps sequential per-shard replay within budget.
4. Extend passivation timeout. Ensure passivate-idle-entity-after exceeds the expected rebalance window; otherwise hot entities passivate mid-rebalance and defeat rememberEntities.
5. Disable replay filter if safe. If your events are deterministic, set akka.persistence.cassandra.journal.replay-filter.mode = off to skip filter overhead.

Verification before rollout

Trace a single journal read to confirm local-DC latency:

cqlsh> TRACING ON;
cqlsh> SELECT * FROM journal WHERE persistence_id = 'entity-123';

Run a staged rebalance with rememberEntities = true, 600 shards, and the snapshot settings above. Capture p99 recovery from ClusterSharding metrics (shard-started, entity-started) and confirm the snapshot store uses the same local DC with write-consistency = LOCAL_QUORUM. Because plugin defaults can shift across patch releases, validate every config key against your exact Akka and Cassandra plugin version before production rollout.

One diagnostic that changes the recommendation

Is your Cassandra deployment single-DC or multi-DC, and are the journal and snapshot tables restricted to local-DC replication? If cross-DC reads are unavoidable, the tuning above may still miss the 800 ms target and you should evaluate a local read replica or a local snapshot cache before increasing snapshot frequency further.