Redis Streams: Designing Reliable Event Processing with Consumer Groups

Requirements: What You Actually Need

Before choosing Redis Streams, verify:

1. Do you need durable, ordered message storage?
2. Do you want multiple consumers to process the same stream with independent positions?
3. Is at‑least‑once delivery acceptable (duplicate processing possible)?
4. Can you tolerate a single Redis instance as the source of truth (or rely on replication for HA)?

Smallest Suitable Design

A single stream named events with a consumer group workers. Producers use XADD to append messages. Consumers claim pending messages with XREADGROUP GROUP workers consumer‑1 COUNT 1 BLOCK 0 STREAMS events > and acknowledge with XACK.

Example Commands

# Producer: add a message
redis-cli XADD events * sensor temperature 23.5

# Consumer: read new messages (replace consumer‑1 with your name)
redis-cli XREADGROUP GROUP workers consumer-1 COUNT 1 BLOCK 0 STREAMS events >

# After processing, acknowledge
redis-cli XACK events workers 

Trust and Data Boundaries

All clients trust the Redis instance completely. In production you should:

• Enable authentication via requirepass or ACLs (Redis 6+).
• Encrypt traffic with TLS (Redis 6+).
• Place Redis inside a private network or VPC.

Operational Checks

Monitor these metrics to gauge health:

• INFO fields stream:num Streams, stream:num Consumers, stream:num Pending.
• Stream length with XLEN events.
• Pending messages per consumer via XPENDING events workers - - + count.
• Replication lag if using replicas (INFO replication).

Monitoring Command

# Stream length and pending summary
redis-cli XLEN events
redis-cli XPENDING events workers - - 100

Failure Modes and When to Change Design

1. Consumer crashes before XACK – message becomes pending and will be redelivered after the visibility timeout (or immediately on next read). No loss, but duplicates possible.
2. Network partition separating producers from Redis – producers receive errors; consider buffering or fallback.
3. Stream trimming policy too aggressive – may delete messages before consumers acknowledge, causing loss.
4. Replication lag in a primary‑replica setup – consumers reading from replicas may see stale data.
5. Need cross‑shard scaling – Redis Cluster does not shard streams; you must route all stream keys to a single hash slot or use an external bus.

When to Consider Alternatives

• Exactly‑once processing required – consider Kafka with idempotent consumers or transactional producers.
• Geographically distributed low‑latency ingestion – look at Pulsar or AWS Kinesis.
• Need to shard streams by key for massive scale – use a partitioned log like Kafka.
• Require complex routing or topic hierarchies – RabbitMQ exchanges may be a better fit.

Conclusion

Redis Streams with consumer groups give you a simple, durable, ordered event log suitable for many intra‑service notification and workflow scenarios, as long as you stay within a single Redis instance (or a replicated primary) and can tolerate at‑least‑once delivery. For stricter ordering, exactly‑once, or multi‑shard scalability, evaluate a dedicated streaming platform.