Fixing Per-Key Ordering in Apache Pulsar with Key_Shared Subscriptions

If you have ever run a Shared subscription in Apache Pulsar to parallelize processing, you have probably discovered the hard way that per-key ordering disappears. The broker hands each message to whichever consumer is next in the round-robin, so two updates for the same user ID can land on different workers and be applied out of sequence. Switching to Failover preserves order, but only one consumer is active at a time, so you lose horizontal throughput. Key_Shared subscriptions exist to close that gap: they keep every message for a given key on a single consumer while still spreading different keys across the fleet.

Why round-robin and failover both disappoint

A Shared subscription maximizes parallelism by distributing messages across all attached consumers with no ordering guarantee. That is fine for independent tasks, but it breaks workflows that depend on sequence—per-account balances, per-device state machines, or ordered event sourcing. A Failover subscription solves ordering by routing everything to one active consumer and keeping the rest as hot standbys. The result is correct but serial: your consumer group is only as fast as one node.

Key_Shared sits between these two modes. The broker hashes each message key onto a consistent hash ring and pins all messages with that key to one consumer. Different keys go to different consumers, so throughput scales with the number of workers, yet any single key stays on one worker and remains ordered.

How the broker routes keys

The mapping is owned by the broker, not the client. When you attach a consumer to a Key_Shared subscription, the broker assigns it a slice of the hash space. Two policies control that assignment:

• autoSplit divides the hash space into equal contiguous ranges, one per consumer, and re-splits the ranges whenever consumers join or leave.
• sticky assigns fixed ranges and moves as few keys as possible when membership changes, which reduces churn but can create imbalance if the key distribution is skewed.

Ordering is per key, not global. The producer sets a key—for example producer.newMessage().key("user-42") in the Java client—and only messages sharing that key are guaranteed in-order delivery. On a partitioned topic the mapping is computed per partition, so the same key in two partitions can be handled by different consumers. Do not assume cross-partition ordering.

Subscription	Ordering	Active consumers	Typical use
Shared	None	All	Independent tasks
Failover	Global	One	Single-worker ordering
Key_Shared	Per key	All	Ordered per entity at scale

A worked multi-consumer example

Imagine an order-processing topic where each message carries an orderId key and a monotonically increasing sequence number. You want three consumers working in parallel, with the ability to scale to five without breaking the sequence for any individual order.

On the producer side, set the key explicitly:

producer.newMessage()
    .key(orderId)
    .value(sequencePayload)
    .send();

On the consumer side, create the subscription with the Key_Shared type and an auto-split policy:

Consumer<byte[]> consumer = client.newConsumer()
    .topic("persistent://my-tenant/my-namespace/orders")
    .subscriptionName("order-processor")
    .subscriptionType(SubscriptionType.Key_Shared)
    .keySharedPolicy(KeySharedPolicy.autoSplit())
    .subscribe();

With three consumers attached, the broker splits the hash ring into three ranges and routes each orderId to exactly one consumer. If you then attach two more consumers, auto-split re-divides the space into five ranges. If you kill one consumer mid-stream, its range is reassigned. In a correct run, each consumer log shows sequence numbers that increase monotonically per key, even as the set of active consumers changes.

Because the subscription type is fixed when the subscription is created, you cannot convert an existing Shared subscription to Key_Shared in place. Adopting it means creating a new subscription name and either replaying the topic or cutting over at a safe boundary.

The batching footgun and rebalancing edges

The most common surprise is producer batching. If a batch contains messages with different keys, Pulsar dispatches that batch as a unit to a single consumer. Same-key messages that happen to be split across batch boundaries can therefore be processed out of order. When strict per-key ordering matters, disable producer batching or ensure every batch contains only one key. Confirm the exact batch-handling behavior for your Pulsar version, because this area has changed across releases.

Key_Shared supports individual acknowledgements only; cumulative acks are not available on shared-style subscriptions. After a nack or an ack timeout, redelivery is designed to return a message to the consumer that owns its key, which protects ordering. However, during a rebalance—when auto-split reassigns ranges because the consumer count changed—unacknowledged or redelivered messages for a moved key may not keep strict order through the transition. Test this with a chaos scenario, not just a steady-state benchmark.

Diagnosing skew and verifying the result

Operationally, you want to see an even spread of work. Run the following on a machine with pulsar-admin configured against your cluster; you need admin or stats-level permissions on the tenant or namespace:

pulsar-admin topics stats \
  --topic persistent://my-tenant/my-namespace/orders

Expected check: the subscription lists the number of consumers you attached, and their unacked-message counts are roughly balanced. The command is read-only, but polling it in a tight loop adds unnecessary load to the broker, so sample it periodically rather than continuously. If one consumer holds a large backlog while others are idle, that usually indicates a hot key rather than a Pulsar defect. Look at the key distribution in your data before blaming the hash ring.

To verify ordering yourself, produce a fixed set of keys with sequence numbers, run two or more Key_Shared consumers, and assert that each key's sequence arrives monotonically. Then add and remove consumers while messages are in flight and in the redelivery queue, and diff the per-key ordering again. That exposes rebalance gaps that a quiet cluster will hide.

When to reach for it

Key_Shared is the right tool when you need ordered processing per entity—per-user state, per-account ledgers, per-device telemetry—and you want that ordering to scale horizontally. It is the wrong tool if you need global ordering, if you rely on cumulative acknowledgements, or if your key distribution is so skewed that one key dominates the topic. Confirm that your broker and client versions expose SubscriptionType.Key_Shared and KeySharedPolicy; the feature became available in the Pulsar 2.8 timeframe, and older deployments will reject the subscription type. Start with auto-split, watch the unacked counts, and keep batches single-key until you have proven your version's batching behavior under failure.