Using Pulsar's KeyShared Subscription for Ordered Per‑Key Processing

Overview

The KeyShared subscription type in Apache Pulsar gives you ordering guarantees for messages that share the same key while still allowing multiple consumers to share the workload. This combines the strict ordering of an Exclusive subscription with the load‑balancing of a Shared subscription.

Configuration Example

Below is a minimal Java consumer that enables KeyShared. Run this code in any Java application that has the Pulsar client library on its classpath. The application needs permission to create a consumer and subscription on the target topic.

import org.apache.pulsar.client.api.Consumer;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.SubscriptionType;

public class KeySharedConsumer {
    public static void main(String[] args) throws Exception {
        String serviceUrl = "";   // e.g. pulsar://localhost:6650
        String topic = "";                     // e.g. persistent://public/default/orders
        String subName = "";      // e.g. order-processor

        try (PulsarClient client = PulsarClient.builder().serviceUrl(serviceUrl).build()) {
            Consumer consumer = client.newConsumer()
                    .topic(topic)
                    .subscriptionName(subName)
                    .subscriptionType(SubscriptionType.KeyShared)
                    .subscribe();

            while (true) {
                consumer.receiveAsync().thenAccept(msg -> {
                    // Process message – ordering is guaranteed per key
                    String key = msg.getKey();
                    System.out.printf("Consumer %s got key=%s%n", Thread.currentThread().getName(), key);
                    consumer.acknowledge(msg);
                }).join();
            }
        }
    }
}

How It Works

When a message is published, the broker computes a hash of the message key (if present) and maps that hash to a specific consumer instance among the subscribers using the same subscription name. All messages with the same key hash are therefore delivered to the same consumer, preserving order for that key. Different keys may be handled by different consumers, enabling parallelism.

Limits and Common Mistakes

• Ordering scope: Guarantees apply only to messages that share an identical key. Messages with different keys have no cross‑key ordering.
• Key skew: If a small set of keys dominates the traffic, the consumer(s) assigned to those keys can become a hotspot, leading to uneven lag.
• Null or missing keys: When a message has a null or empty key, the broker falls back to Shared semantics, losing per‑key ordering.
• Broker overhead: The broker must hash the key for each message, adding a tiny latency cost (typically sub‑millisecond).

Verification Steps

To confirm that ordering per key is working:

1. Produce a test stream with a known key pattern, e.g., user-{i%10} for i = 0..999.
2. Start two or more Java consumers using the code above with the same subscription name.
3. Observe the console output: all messages with the same numeric suffix should be printed by the same consumer thread.
4. Check broker stats via pulsar-admin persistent stats --subscription or Pulsar Manager; look for balanced consumer lag per key and absence of duplicate processing messages in the logs.

If you notice a single consumer handling most of the traffic, inspect the key distribution; consider re‑keying (e.g., using a composite key) or splitting the topic to alleviate hotspots.