Avoiding Hot‑Spot Shards in Akka Cluster Sharding

Concrete Problem: Hot‑Spot Shards Drain Your Cluster

When you run a stateful service in Akka and enable ClusterSharding, you expect actors to be distributed across nodes automatically. In practice, a handful of shards can absorb most of the traffic. The result is a few overloaded nodes while others sit idle. The root cause? Poorly distributed entity IDs that map to the same shard IDs.

Thesis: Design Your Entity IDs and Extractor to Spread Load

Akka’s sharding mechanism relies on a hash of the entity ID to pick a shard. If the hash space is unevenly used, you get “hot” shards. By choosing IDs that produce a uniform hash distribution and by configuring the extractor appropriately, you can keep shards balanced and avoid performance bottlenecks.

1. How Cluster Sharding Maps Actors to Shards

ClusterSharding uses a MessageExtractor that implements three methods:

• entityId(message) – extracts the entity ID from a message.
• shardId(entityId) – turns an entity ID into a shard ID, usually by hashing.
• entityMessage(message) – returns the message to forward to the entity actor.

The default HashCodeMessageExtractor uses entityId.hashCode % numberOfShards. The ClusterShardingMediator then routes the message to the node that owns the shard. When a node joins or leaves, the mediator recalculates shard ownership, potentially creating actors on new nodes.

2. Design Entity IDs for Uniform Hashing

Sequential or predictable IDs (e.g., user-1, user-2…) produce similar hash codes. Even a small change in the string can cause many IDs to map to the same shard. Instead, consider:

• Prefix the ID with a random or hash‑based component: sha1(userId) + -user.
• Use UUIDs: UUID.randomUUID() guarantees a uniform distribution.
• If you must keep human‑readable IDs, add a hash prefix: hash(userId)-userId.

Example: entityId = "${userId.hashCode()}-${userId}" ensures that the hash influences the shard choice while the original ID remains readable.

3. Configure the Extractor and Shard Count

Define a custom extractor in Scala:

import akka.cluster.sharding.typed.scaladsl.{ClusterSharding, ClusterShardingSettings}
import akka.cluster.sharding.typed.scaladsl.ShardRegion.{MessageExtractor, ShardId}

class UserMessageExtractor(numberOfShards: Int) extends MessageExtractor {
  override def entityId(message: Any): String = message match {
    case UserCommand(id, _) => s"${id.hashCode()}-$id"
    case _ => throw new IllegalArgumentException("Unknown message")
  }

  override def shardId(entityId: String): ShardId = {
    val hash = entityId.hashCode
    (math.abs(hash) % numberOfShards).toString
  }

  override def entityMessage(message: Any): Any = message
}

Register it with ClusterSharding:

val settings = ClusterShardingSettings(system).withNumberOfShards(200)
ClusterSharding(system).init(
  Entity(typeName = "UserEntity"){ entityContext =>
    UserActor(entityContext.entityId)
  }
  .withMessageExtractor(new UserMessageExtractor(200))
)

Key points:

• Choose numberOfShards significantly larger than the expected number of entities to reduce collision probability.
• Ensure entityId is unique per actor; otherwise, Akka will treat them as the same actor.
• Run the extractor locally to verify that shardId distribution is even.

4. Verify Distribution in a Test Cluster

Deploy a minimal three‑node cluster and send a burst of messages:

1. Start three seed nodes with akka.cluster.seed-nodes configured.
2. Send 10,000 UserCommand messages with random user IDs.
3. Use ClusterSharding.get(system).shardRegion("UserEntity") and call tell(ShardRegion.GetShardRegionStats, Actor.noSender) to retrieve stats.
4. Inspect the shardToRegion map; each shard should be on a different node and the load per node should be roughly equal.

Risks to watch for:

• During node churn, shards may be temporarily unassigned, causing duplicate actor creation.
• If the extractor is misconfigured, you could end up with empty shards and wasted capacity.

5. Trade‑offs and Limitations

Increasing numberOfShards improves distribution but adds overhead: more region actors and larger routing tables. If you set it too high, you may hit the akka.cluster.sharding.max-shard-count limit or consume excessive memory.

Also, Akka cannot re‑hash existing shards on the fly. If you discover a hotspot after deployment, you must restart the cluster with a new extractor or adjust the number of shards, which will cause a temporary service disruption.

6. Actionable Checklist

• Choose a hash‑based or UUID entity ID strategy.
• Set numberOfShards to at least 3× the expected active entities.
• Implement a MessageExtractor that separates entityId and shardId cleanly.
• Deploy a test cluster and run ShardRegion.GetShardRegionStats to confirm even distribution.
• Monitor ClusterShardingMediator logs for duplicate actor creation during churn.
• Plan for a controlled restart if you need to change shard count after deployment.

By paying attention to entity ID design and extractor configuration, you can keep Akka Cluster Sharding from creating bottlenecks and ensure that your stateful actors scale gracefully across the cluster.