Building Reliable Producer‑Consumer Pipelines with Redis Streams

The Problem: Lost Messages and Scaling Consumers

Many applications need a durable way to pass data from producers to multiple consumers while tolerating occasional processing failures. Simple pub/sub drops messages if a consumer is offline, and home‑grown queues often require complex bookkeeping to guarantee at‑least‑once delivery. As the number of consumers grows, managing separate queues or polling mechanisms becomes operationally heavy.

How Redis Streams Solve It

Redis Streams, introduced in Redis 5.0, model an append‑only log where each entry has a globally unique ID (timestamp‑sequence) and a map of field‑value pairs. Consumer groups allow several clients to cooperate on the same stream: each new message is delivered to one consumer in the group, and the group tracks which messages have been processed but not yet acknowledged via a Pending‑Entries List (PEL). This gives at‑least‑once semantics without requiring each consumer to maintain its own offset.

Worked Example: Setting Up a Stream with Consumer Group

The following commands illustrate a minimal producer‑consumer flow. Run them in a redis-cli session connected to a Redis server (version 5.0 or later). No special privileges are required beyond the ability to write to the database; however, be aware that uncontrolled growth of the PEL can exhaust memory.

1. Create a stream and add a few entries (the producer side).

   XADD mystream * sensor-id 123 temperature 22.5
   XADD mystream * sensor-id 124 temperature 23.0
   XADD mystream * sensor-id 125 temperature 21.8
   

2. Create a consumer group named mygroup starting at the beginning of the stream.

   XGROUP CREATE mystream mygroup $ || true
   

   The || true prevents an error if the group already exists.

3. Have a consumer (consumer1) read new messages.

   XREADGROUP GROUP mygroup consumer1 COUNT 1 BLOCK 0 STREAMS mystream >
   

   This blocks until a message is available, returns the entry, and leaves it in the PEL.

4. Acknowledge processing.

   XACK mystream mygroup 
   

   Replace <message-id> with the ID returned by the XREADGROUP call. After acknowledgment, the entry disappears from the PEL.

5. Inspect pending entries to verify the group’s state.

   XPENDING mystream mygroup - + 10 consumer1
   

   If the consumer crashes before XACK, the entry remains in the PEL and will be redelivered to another consumer (or the same one after a timeout).

6. Apply approximate trimming to keep memory usage bounded.

   XADD mystream MAXLEN ~ 1000 * sensor-id 126 temperature 22.9
   

   The tilde (~) tells Redis to perform an efficient, approximate trim; you can check the resulting length with XLEN mystream.

Trade‑offs and Limitations

• Pending‑Entries List growth: If consumers fail to acknowledge messages (due to bugs, crashes, or network partitions), the PEL can accumulate indefinitely, causing memory pressure. Monitoring XPENDING length and setting a consumer‑side timeout or dead‑letter process is essential.
• Approximate trimming may discard needed data: Using MAXLEN ~ N removes old entries to stay near N, but the exact cutoff is not guaranteed. Late‑joining consumers that rely on replaying historic events may miss data; consider persisting a separate replay buffer or using a longer MAXLEN if replay is required.
• NOACK lowers latency but risks loss: The XREADGROUP … NOACK variant skips explicit acknowledgments, improving throughput at the cost of at‑least‑once guarantees. Use it only when occasional message loss is acceptable.

Actionable Checklist

1. Verify Redis version: redis-server --version (≥5.0).
2. Create streams and consumer groups with XGROUP CREATE (idempotent via || true).
3. Always acknowledge processed messages with XACK unless you explicitly accept loss via NOACK.
4. Monitor PEL size regularly: XPENDING - - 100 and alert if it trends upward.
5. Set trimming based on your replay needs: XADD … MAXLEN ~ and confirm with XLEN.
6. Test failure scenarios: kill a consumer after XREADGROUP but before XACK and observe redelivery via another consumer or the same consumer after restart.