Choosing the Right Apache Pulsar Subscription Mode for Consumer Scaling

The Scaling Dilemma: Ordering vs. Throughput

When designing a consumer application in Apache Pulsar, the primary engineering trade-off is between strict message ordering and horizontal processing throughput. Choosing the wrong subscription mode can lead to either a processing bottleneck where a single consumer cannot keep up with the ingress rate, or data corruption where messages are processed out of sequence.

The decision hinges on whether your business logic requires a global sequence of events (e.g., a financial ledger) or if events are independent and can be processed in parallel (e.g., sending notification emails).

Subscription Mode Comparison

Mode	Ordering Guarantee	Scaling Method	Primary Use Case
Exclusive	Strict Global	None (Single Consumer)	Single-threaded state machines
Failover	Strict Global	Active-Passive Standby	High-availability sequential processing
Shared	None	Round-Robin Distribution	High-throughput worker queues
Key_Shared	Per-Key Ordering	Key-based Hashing	Partitioned stateful processing

Engineering Trade-offs

Exclusive and Failover: Prioritizing Sequence

Exclusive mode allows only one consumer to attach to a subscription. If a second consumer attempts to connect, it is rejected. Failover mode improves on this by allowing multiple consumers to attach, but only one (the master) receives messages. If the master disconnects, Pulsar automatically promotes a standby consumer.

These modes are essential when the order of operations is critical. However, they create a processing ceiling: your maximum throughput is limited to the capacity of a single consumer instance.

Shared and Key_Shared: Prioritizing Throughput

Shared subscriptions distribute messages across all attached consumers. This allows you to scale linearly by adding more pods or instances. The risk is that messages are processed concurrently, meaning message B might finish processing before message A, even if A was published first.

Key_Shared solves the "ordering vs. scale" conflict by ensuring that all messages with the same ordering key are routed to the same consumer. This allows for parallel processing across different keys while maintaining strict sequence for any individual entity (e.g., all events for User_123 always go to Consumer A).

Implementation Example: Key_Shared Configuration

To implement a Key_Shared subscription using the Java client, you must specify the subscription type during the consumer creation process. This assumes you are using Pulsar 2.x or later.

// Run this within your consumer application logic
// Required Permissions: Client must have 'consume' permission on the topic
Consumer<byte[]> consumer = pulsarClient.newConsumer()
    .topic("persistent://public/default/my-topic")
    .subscriptionName("my-scaling-subscription")
    .subscriptionType(SubscriptionType.KeyShared)
    .subscribe();

Validation and Diagnostics

To verify that your subscription mode is behaving as expected, use a multi-consumer test environment:

1. For Shared/Key_Shared: Start three consumer instances. Produce 100 messages. Check the logs to ensure messages are distributed across all three instances rather than being trapped by one.
2. For Failover: Start two consumers. Verify only one receives messages. Kill the primary process and monitor the logs of the second consumer to see the handover (failover) occur.
3. For Key_Shared Ordering: Produce messages with keys K1, K2, K1, K2. Verify that all K1 messages are handled by the same consumer instance and in the order they were sent.

Limitations and Risks

• Subscription Migration: Changing a subscription type on an existing subscription name can be disruptive. It is recommended to create a new subscription name when changing the scaling strategy to avoid unexpected message redelivery or client errors.
• Key Distribution: In Key_Shared mode, if a disproportionate number of messages share the same key (a "hot key"), one consumer will be overloaded while others remain idle, negating the benefits of horizontal scaling.
• Failover Latency: Failover is not instantaneous. There is a brief window during the heartbeat timeout where no messages are processed.

Rollback Procedure

If a subscription mode change causes processing errors (e.g., out-of-order data in a Shared subscription), perform the following:

1. Stop all active consumers for that subscription.
2. Update the client configuration to the previous SubscriptionType.
3. Restart consumers. Note that messages already acknowledged in the "wrong" mode cannot be automatically rolled back; you may need to reset the subscription cursor using the Pulsar admin CLI: pulsar-admin topics reset-cursor persistent://public/default/my-topic --subscription my-scaling-subscription.