Processing Redis Streams with Consumer Groups for Exactly‑Once Delivery

Useful answer

When you need multiple workers to process a stream of events while guaranteeing that each event is handled exactly once, use a Redis Streams consumer group. The group tracks which messages have been delivered but not yet acknowledged, allowing failed workers to reclaim pending messages and preventing duplicate processing.

Worked example

Run the following commands in a redis-cli> prompt (or any Redis client) against a Redis 7+ server. You need permission to execute stream commands (default for authenticated users). Replace placeholders with your own values.

# 1. Add a few sample messages to a stream
XADD mystream * sensor temperature 23.5
XADD mystream * sensor temperature 24.0
XADD mystream * sensor temperature 22.8

# 2. Create a consumer group; start reading from the newest message ($)
XGROUP CREATE mystream mygroup $  # returns OK if group does not exist

# 3. In terminal A, simulate consumer A
XREADGROUP GROUP mygroup consumerA COUNT 1 BLOCK 0 STREAMS mystream >
#   -> returns one message, e.g., 1729000000000-0 sensor temperature 23.5

# 4. In terminal B, simulate consumer B (run the same XREADGROUP command)
XREADGROUP GROUP mygroup consumerB COUNT 1 BLOCK 0 STREAMS mystream >
#   -> returns a different message, e.g., 1729000000001-0 sensor temperature 24.0

# 5. After processing each message, acknowledge it with XACK
XACK mystream mygroup 1729000000000-0   # run in terminal A after its work
XACK mystream mygroup 1729000000001-0   # run in terminal B after its work

# 6. Verify that no pending messages remain
XPENDING mystream mygroup - - 100
#   -> should show 0 pending entries

Where to run: any host with network access to the Redis instance; the commands are executed via redis-cli or a language‑specific Redis client. Required permissions: ability to execute XADD, XGROUP, XREADGROUP, and XACK (default for authenticated users).

Limits and common mistakes

• Pending list memory usage: Consumer groups keep unacknowledged message IDs in memory. If workers crash or forget to XACK, the pending list grows and can consume significant RAM. There is no hard constant limit; the bound is the available memory.
• Regular acknowledgment: Always call XACK after successfully processing a message. Forgetting to do so leads to redelivery on worker restart and steady memory growth.
• Stream trimming: Producers may add messages faster than consumers can acknowledge them. Use XTRIM or the MAXLEN option on XADD to limit stream length and prevent unbounded growth.
• Using the wrong read command: XREAD bypasses consumer‑group semantics and delivers every message to each caller, breaking the exactly‑once guarantee. Use XREADGROUP for group‑based consumption.
• Group creation after data exists: If you create a group after producers have already added messages, you must choose a start ID. Use $ to receive only new messages, or 0 to process the entire backlog.
• Handling worker failures: When a worker dies, its pending messages become eligible for reclamation via XCLAIM after a configurable idle time. Design your workers to periodically call XCLAIM or rely on a supervisor that does so.

How to verify the result

After processing and acknowledging messages, run:

XPENDING   - - 100

A pending count of zero indicates all delivered messages have been acknowledged. You can also monitor memory usage with:

INFO memory

Look for the used_memory value; it should not show a steady upward trend when acknowledgments are timely. If you observe growth, check for missing XACK calls or consider increasing the rate of acknowledgments.