Key_Shared in Apache Pulsar: per-key ordering without partition math

An order pipeline is the classic trap. Every order emits created, paid, and shipped events to one Pulsar topic. A single consumer preserves the sequence but falls behind at peak. Switch to a Shared subscription and the load spreads out — but paid can reach a different consumer and get processed before created ever arrives. Partitioning the topic fixes ordering, but now your maximum parallelism equals your partition count, and repartitioning a live topic is a migration project.

Key_Shared is Pulsar’s middle path: a subscription type that delivers every message with the same key to the same consumer, while different keys spread across the whole consumer group. You get per-entity ordering and horizontal scaling without needing one partition per consumer. The costs are real but narrow: hot keys, and a short reorder window whenever consumers join or leave.

This post assumes a Pulsar 2.10-or-newer broker. Key_Shared has existed since 2.4, but range‑assignment defaults and broker settings have shifted between releases, so treat the specifics here as things to confirm against the version you actually run.

How Key_Shared routes a message

When a producer attaches a key to a message, the broker hashes that key into a numeric range. The key space is divided into ranges, and each connected consumer owns a slice of them. Any message whose hash falls inside a consumer’s slice goes to that consumer, in the order the broker received it. Different keys hash into different slices, so the group processes them concurrently.

Range ownership works in two modes: auto‑split, where ranges are redistributed when consumers join or leave, and sticky, where each consumer holds a fixed set of ranges. Which mode is the default, and how it is configured, has changed across releases — the relevant broker settings are grouped under names beginning with subscriptionKeyShared, so check your version’s documentation rather than copying config from an older article (this one included).

For context, here is where Key_Shared sits among Pulsar’s four subscription types:

One contrast worth spelling out: Kafka‑style consumer groups tie parallelism to partition count, because the partition is the unit of assignment. Pulsar decouples subscriptions from partitions — you scale a Key_Shared subscription by starting another consumer, not by adding partitions.

A worked example: one order topic, three consumers

Create a test topic. Run pulsar-admin from a machine that can reach the broker’s HTTP service (port 8080 by default) with a role permitted to create topics in the namespace — swap in your own tenant/namespace in place of public/default. If auto‑topic creation is enabled on your broker, producing to the topic also creates it:

pulsar-admin topics create persistent://public/default/orders-demo

Publish with the order ID as the message key. In the Java client:

Producer<byte[]> producer = client.newProducer()
    .topic("persistent://public/default/orders-demo")
    .create();

producer.newMessage()
    .key("order-42")
    .value("{\"order_id\":\"order-42\",\"event\":\"shipped\"}")
    .send();

Then run three consumer instances that share a subscription name and declare the Key_Shared type:

Consumer<byte[]> consumer = client.newConsumer()
    .topic("persistent://public/default/orders-demo")
    .subscriptionName("order-processing")
    .subscriptionType(SubscriptionType.Key_Shared)
    .subscribe();

Interleave a few events for order-42 and order-17, then check the consumer logs. What you want to confirm: every event for a given order arrives at exactly one consumer, in publish order, while the two orders may be handled by different consumers. If a key’s events show up across consumers while the group is stable, the usual culprit is a producer publishing without a key — keyless messages get no per‑key ordering guarantee, so don’t mix keyed and keyless traffic on a subscription that needs ordering. If you need to route by one field but order by another, newer clients offer a separate ordering‑key field; verify your client version supports it before relying on it.

The test topic does change broker state, so clean it up when finished:

pulsar-admin topics delete persistent://public/default/orders-demo

The two costs: hot keys and rebalance windows

Hot keys. All traffic for one key is pinned to one consumer. If a single tenant or a flash‑sale order generates a torrent of events, one consumer absorbs all of it no matter how large the group grows. No broker setting fixes a skewed key — the fix is the key choice itself. Prefer high‑cardinality, evenly distributed keys (an order ID rather than a tenant ID when one tenant dominates), and if the natural key is skewed, either accept the imbalance or split the hot key into subkeys at the cost of ordering across those subkeys.

Rebalance windows. When a consumer joins or leaves, ranges are reassigned, and a key that moves to a new consumer has its ordering boundary at the handoff. Brief out‑of‑order delivery for migrated keys is possible, particularly under auto‑split. Design handlers to tolerate it — idempotent processing, or a per‑key sequence‑number check — and consider sticky ranges if your group membership churns. Either way, don’t assume ordering survives a membership change; test it.

Acknowledgment. Shared and Key_Shared subscriptions do not support cumulative acknowledgment (acking every message up to a given one at once). Each message must be acknowledged individually, which changes redelivery semantics and adds ack traffic. If you are migrating from Failover code that acks cumulatively, that code needs updating before the switch.

Before you point production at it

Verify on your own cluster, since defaults are version‑sensitive: publish keyed messages from several keys, attach two or three consumers on a Key_Shared subscription, and confirm each key’s events land on one consumer in order. Then stop a consumer and start another mid‑run, and watch whether any migrated key reorders. Check your broker version’s documentation for the active range‑assignment mode and its settings.

The decision rule: per‑entity ordering with an elastic consumer count calls for Key_Shared; strict order across the whole topic calls for Exclusive or Failover; no ordering requirement means Shared is fine. And if you find yourself adding partitions purely to add consumers, that is exactly the problem Key_Shared exists to remove.