Redis Streams Consumer Group Guide

Learn how to create a Redis Streams consumer group, consume messages, acknowledge them, monitor pending entries, and recover from consumer failures using XCLAIM.

Desired Outcome

Configure a Redis Streams consumer group so that messages are processed at least once, can be acknowledged safely, and can be reclaimed by another consumer if the original processor fails. The guide assumes you will use redis-cli for demonstration, but the same commands apply to any client library that supports the Streams API.

Prerequisites

Redis server version 5.0 or newer (consumer groups were introduced in Redis 5.0).
Access to a terminal with redis-cli or a Redis client library that implements XADD, XGROUP, XREADGROUP, XACK, XPENDING, and XCLAIM.
Basic knowledge of Redis command syntax (key names, arguments, and optional flags).

Procedure

Create the stream and consumer group
If the stream does not already exist, create it together with the consumer group in one command. The special ID $ means the group starts receiving only new messages.
```
XGROUP CREATE mystream mygroup $ MKSTREAM
```
Run this command once; subsequent executions will return an error if the group already exists, which is safe to ignore.
Add one or more consumers
Each consumer needs a unique name within the group. You can think of the name as a worker identifier.
```
# Example consumer names: worker1, worker2
# No explicit command is needed; the name is supplied when reading.
```
Consume messages
Each consumer repeatedly runs a blocking read to fetch new messages, processes them, and then acknowledges.
```
XREADGROUP GROUP mygroup worker1 COUNT 1 BLOCK 0 STREAMS mystream >
```
The command returns a message ID and its fields. Process the payload according to your application logic.
Acknowledge processing
After successful processing, tell Redis that the message is handled so it is removed from the pending list.
```
XACK mystream mygroup 
```
Replace <message-id> with the ID returned by the read step.
Monitor pending messages
Check which messages have been delivered but not yet acknowledged.
```
XPENDING mystream mygroup - + 10
```
The output shows up to 10 pending entries with details such as consumer name, idle time, and delivery count.
Recover from a failed consumer
If a consumer crashes, its unacknowledged messages stay in the pending list. Another consumer can claim them using XCLAIM.
```
XCLAIM mystream mygroup claimer-worker 0 
```
The idle time (0 in the example) specifies the minimum idle period; setting it to 0 claims immediately. The claiming consumer then processes the message and must call XACK afterward.

Expected Checks

After creating the group, run XINFO GROUPS mystream to verify that mygroup appears with a consumer count of 0 (or the number you have added) and a pending count of 0.
While a consumer is active, execute the read command, confirm that the returned message ID matches one you added via XADD, then run XACK and re‑run XPENDING to see the pending count decrease by one.
To test recovery, stop a consumer process, then run XCLAIM on one of its pending message IDs. Verify that the claiming consumer can read the message again and that after processing and acknowledging, the pending count drops.

Recovery Options and Limitations

Message loss risk: A message is considered processed only after XACK. Forgetting to acknowledge leaves it in the pending list, causing redelivery after the consumer’s visibility timeout (implicitly the idle time you set in XCLAIM).
Consumer lag: If processing is slower than the arrival rate, the pending list can grow. Monitor the length returned by XPENDING and consider adding more consumers or optimizing the processing logic.
Version compatibility: Consumer groups require Redis 5.0+. Older servers will return an unknown command error for XGROUP, XREADGROUP, etc.
Idempotency: Because a message may be delivered more than once (e.g., after a claim), design your processing logic to be idempotent or detect duplicates using application‑level identifiers.

Practical Verification

After completing the steps above, you can confirm end‑to‑end reliability with this sequence:

Add a test message: XADD mystream * foo bar (note the generated ID).
Have a consumer read and process it, then XACK.
Run XPENDING mystream mygroup - + 10 and ensure the pending count is 0.
Kill the consumer, run XCLAIM mystream mygroup claimer-worker 0 <id-from-step-1>, have another consumer read the claimed message, process it, and XACK again.
Verify pending count returns to 0.

If any step shows a non‑zero pending count where you expect zero, check that the corresponding XACK was executed and that no consumer is holding the message without acknowledging.

Setting Up Redis Streams Consumer Groups for Reliable Message Processing