Akka Cluster Sharding: remember-entities vs passivation-only for sub-second recovery SLO

Context

We operate a 6-node Akka 2.9 cluster (Scala 3.3) hosting roughly 120,000 sharded entities of a single type. Each entity holds ~5 KB of event-sourced state. Our SLO requires p99 entity availability within 800 ms after a node loss or shard rebalance.

Constraint

The rememberEntities flag is entity-type global and immutable without a full rolling restart. Enabling it persists shard-to-entity mappings to our Cassandra journal, guaranteeing automatic restart but adding journal read + event replay latency (observed 1.2–3 s per entity in staging). Passivation-only avoids this latency but drops passivated entities on rebalance; recovery then depends on external traffic rehydrating each entity via its command handler, which is non-deterministic and can exceed the SLO during burst rebalances.

Uncertainty

No built-in hybrid mode exists (e.g., TTL-backed persistence for recently active entities only). We could split the workload into two entity types—one with rememberEntities=true for hot entities, one without for cold—but that introduces routing complexity and duplicate sharding infrastructure.

What is the recommended pattern to meet a sub-second recovery SLO for a large, mixed-temperature entity population without operating two separate sharding topologies? Can the journal read path be optimized (e.g., Cassandra local reads, snapshot frequency) to bring rememberEntities recovery under 800 ms at this scale? Are there documented cases of using Akka Distributed Data or a custom ShardAllocationStrategy to approximate hybrid behavior?