Configure RabbitMQ Streams for High-Throughput Ordered Processing

Desired Outcome

Set up RabbitMQ Streams to handle high-throughput message delivery with strict ordering guarantees and credit-based flow control. This configuration supports scenarios like event sourcing, log aggregation, or financial transaction processing where message order and throughput matter.

Prerequisites

• RabbitMQ 3.9 or later installed
• rabbitmq_stream plugin enabled
• Administrative access to the RabbitMQ instance
• Basic understanding of stream concepts (segments, offsets, consumer groups)

Enable the Stream Plugin

Run this command on the server where RabbitMQ is installed:

rabbitmq-plugins enable rabbitmq_stream

Expected check: Verify the plugin is active with rabbitmq-plugins list | grep stream. You should see [E] rabbitmq_stream indicating it's enabled.

Create a Stream with Retention Policy

Streams store messages in an append-only log with configurable retention. Create a stream that retains messages for 24 hours or 1GB, whichever comes first:

rabbitmq-stream create orders-stream \
  --arguments '{"max-age-bytes": 1073741824, "max-age-seconds": 86400}'

Expected check: List streams with rabbitmq-stream list to confirm orders-stream appears with the correct retention settings.

Configure Segment Size for Performance

Segments are the physical files that store stream data. Smaller segments reduce memory usage but may increase I/O during rollover. For most workloads, 10MB segments work well:

rabbitmq-stream update orders-stream \
  --arguments '{"max-segment-size-bytes": 10485760}'

Expected check: Monitor the stream storage directory (typically /var/lib/rabbitmq/mnesia/rabbit@hostname/stream-sockets/) to see segment files being created with appropriate sizes.

Publish Messages with Producer Confirms

Use the RabbitMQ Stream client library to publish messages. Enable producer confirms to ensure messages are durably written:

// Example using Java client
StreamProducer producer = StreamProducer.create(client, "orders-stream");
producer.send(message, SendOptions.builder().producerAcknowledgementMode(ACK).build());

Expected check: Messages should appear in the stream and persist across broker restarts.

Consume with Credit-Based Flow Control

Create a consumer group that processes messages with controlled batch sizes. Credit-based flow prevents consumers from being overwhelmed:

// Example using Java client
StreamConsumer consumer = StreamConsumer.create(client,
  "orders-stream",
  "order-processors",
  ConsumerConfig.builder()
    .offsetSpecification(OffsetSpecification.atStartOfStream())
    .consumerConfiguration(ConsumerConfigParams.consumerConfiguration()
      .initialCreditWindowSize(100))
    .build());

Expected check: Monitor consumer throughput and memory usage. The consumer should process 100 messages at a time, requesting more credit after each batch completes.

Verify Ordered Processing

Each consumer group maintains its own offset position, allowing parallel processing while preserving order within each group. To verify:

1. Publish messages with sequential IDs (1, 2, 3, ...)
2. Have consumers record received IDs in order
3. Confirm no gaps or reordering within each consumer's sequence

Expected check: Consumer logs should show messages processed in strict ascending order by ID.

Monitor and Tune

Key metrics to monitor:

• Segment rollover frequency: Frequent rollovers may indicate segment size is too small
• Consumer lag: Use rabbitmq-stream query consumer-group-lag to check if consumers are keeping up
• Memory usage: Credit window size directly impacts memory consumption

Recovery Options

If configuration changes are needed:

1. Update stream arguments: Use rabbitmq-stream update to modify retention or segment settings
2. Reset consumer offset: Consumers can restart from any offset using OffsetSpecification.atOffset(offset)
3. Delete and recreate stream: For complete reset, use rabbitmq-stream delete followed by creation with new parameters

Risk: Deleting a stream permanently removes all messages. Always ensure you have backups or can tolerate data loss before deletion.