Pulsar Key_Shared Subscriptions: Ordered Per-Key Processing Without Giving Up Parallelism

The problem: order matters, but so does throughput

Imagine an order-processing pipeline. Every status change for an order — created, paid, packed, shipped — lands on a Pulsar topic as a separate message. Your consumers must process events for a single order in sequence, or you risk shipping an order that was never paid for. But you also need to spread work across several consumer instances, because one consumer can't keep up.

Pulsar's standard subscription types force an awkward choice. Exclusive and Failover give you ordering but only one active consumer. Shared gives you parallelism but round-robins messages across consumers, so two events for the same order can be processed concurrently — or out of order.

Key_Shared exists precisely for this gap: multiple consumers on one subscription, with a guarantee that all messages carrying the same key are routed to the same consumer. Per-key ordering, fleet-wide parallelism.

How Key_Shared routes messages

When you produce a message, you set a key (for example, the order ID). The broker's dispatcher maps keys to consumers, so every message for order ord-1042 always goes to the same consumer instance. Different keys land on different consumers, so unrelated orders process concurrently.

Pulsar offers two dispatch modes for Key_Shared:

• Auto-hash — the broker applies consistent hashing to the key and maps hash ranges across consumers. This is the default and the simplest to operate.
• Sticky — you assign explicit key hash ranges to consumers. This gives you control over placement but makes you responsible for range management when the consumer group changes.

Either way, the guarantee is the same: within a stable consumer group, one key maps to one consumer, and messages for that key are delivered in publish order. Acknowledgment is individual (per message), and negatively acknowledged messages are redelivered respecting the key routing — they go back to the consumer that owns the key, not to a random one.

A worked example: order events across three consumers

On the consumer side (Java client, run wherever your consumer service lives; no special broker permissions beyond normal consume rights on the topic):

Consumer<byte[]> consumer = client.newConsumer()
    .topic("persistent://orders/prod/status-events")
    .subscriptionName("order-processor")
    .subscriptionType(SubscriptionType.Key_Shared)
    .subscribe();

On the producer side, always set the key:

producer.newMessage()
    .key(orderId)            // e.g. "ord-1042"
    .value(payload)
    .send();

With three consumers on the order-processor subscription, all events for ord-1042 flow sequentially through one consumer while ord-2087 and ord-3391 are handled in parallel by the others. Scale from three consumers to five and the hash ring redistributes keys automatically.

To verify the routing behaves as promised, run a standalone Pulsar instance locally (bin/pulsar standalone from a Pulsar distribution), produce a few hundred messages with a small set of repeating keys, and log message.getKey() alongside the consumer name in each of two consumers. Every occurrence of a given key should appear in exactly one consumer's log, in order. Then kill one consumer mid-test and watch the keys re-route to the survivor — this also shows you the rebalancing behavior described below.

The trade-offs you need to design around

Hot keys. All messages for one key go to one consumer. If 40% of your traffic is a single key (a celebrity user, a default key, a bug that leaves the key blank), one consumer drowns while the others idle. Choose high-cardinality, roughly uniform keys — order IDs and user IDs usually work; tenant IDs often don't. You can check for skew by comparing backlog and message rates per consumer on the subscription; a badly skewed key shows up as one consumer saturated while others have spare capacity.

Rebalancing breaks ordering briefly. When a consumer joins or leaves, keys move. In-flight and negatively acknowledged messages for a moved key can be redelivered to the new owner before the old owner's processing fully drains, so per-key ordering is not guaranteed during the transition. If your handler is idempotent or version-checked (for example, ignore an event whose sequence number is lower than the last one applied for that order), this window is harmless. If it isn't, this is the risk to engineer for.

No global ordering. Key_Shared guarantees order within a key only. If you need a total order across the whole topic, you're back to Exclusive or Failover and their throughput ceiling.

Version drift. Dispatcher options, defaults, and sticky-range APIs have changed across Pulsar releases. Confirm the Key_Shared section of the documentation for your exact broker version before relying on any specific configuration name or default.

Deciding whether to use it

Reach for Key_Shared when your natural unit of ordering (an order, an account, a device) is much smaller than your total traffic, your keys are high-cardinality and evenly distributed, and your consumers can tolerate brief reordering during scale events. Skip it when you need global ordering, when traffic is dominated by a few keys, or when you can't make handlers idempotent. A quick local test — repeating keys, two consumers, one mid-test restart — will tell you more about fit for your workload than any benchmark table, and it takes less than an hour to run.