Harnessing Redis Streams for Real‑Time Event Processing

Problem: High‑Velocity Event Streams in Modern Apps

Modern services—think IoT dashboards, real‑time analytics, or micro‑service event buses—often need to ingest thousands of events per second, keep them durable, and deliver them to one or many consumers. Traditional queues (e.g., RabbitMQ) are great for work‑distribution, but they drop messages once acknowledged. If a consumer crashes, the data is lost. Persisting the stream of events while still enabling parallel consumption is a common pain point.

Thesis: Redis Streams as a Fault‑Tolerant, Consumer‑Group‑Based Log

Redis Streams (added in Redis 5.0) solve this by combining an append‑only log with consumer groups. Each stream entry is a key/value map, so you can store structured payloads directly in Redis. Consumer groups track per‑consumer offsets, allowing you to pause, resume, or redistribute work without reprocessing already‑handled entries. Because the stream is persisted on disk (AOF or RDB), you get durability, and because Redis is in‑memory, you get sub‑millisecond latency.

Concrete Example: Producing and Consuming Events

1. Start Redis
   

   docker run -d --name redis-streams -p 6379:6379 redis:7
   

2. Produce a sample event
   

   # Connect to the instance
   redis-cli
   # Add an event to "sensor_data"
   XADD sensor_data * temperature 22 humidity 45
   # Expected reply:  (e.g., 1708072212345-0)
   

3. Create a consumer group
   

   XGROUP CREATE sensor_data mygroup $  # $ means start after the latest entry
   # Expected reply: OK
   

4. Consume events with a consumer
   

   XREADGROUP GROUP mygroup consumer1 COUNT 10 BLOCK 0
   # Expected reply: a nested array of entries.
   # Each entry looks like [id, [field, value, field, value, ...]]
   

5. Acknowledge processed entries
   

   XACK sensor_data mygroup 1708072212345-0
   # Expected reply: number of entries acknowledged (e.g., 1)
   

With this minimal setup you already have a durable, parallel‑safe event pipeline. If you add another consumer named consumer2, the stream will split the workload automatically.

Trade‑offs and Limitations

• Memory consumption: Streams are stored in memory. Without trimming, a high‑throughput stream can grow unbounded. Use XTRIM or the MAXLEN option in XADD to bound size:

  XADD sensor_data MAXLEN 10000 * temperature 22
  # Keeps only the most recent 10,000 entries.
  

  If consumers lag behind, you risk dropping unprocessed data.
• Consumer group management: A consumer that fails to acknowledge entries will leave them in the pending entries list (PEL). If the group is rebalanced, those entries may be redelivered, causing duplicates. Use XAUTOCLAIM to claim and reprocess stale entries, or design idempotent consumers.
• Persistence overhead: AOF or RDB persistence adds I/O. For very high write rates, you may need to tune appendfsync or use no-appendfsync-on-rewrite to reduce latency at the cost of durability.
• Single point of failure: A bare Redis instance is a single point of failure. Deploy Redis Sentinel or Cluster for high availability. In a cluster, streams are sharded by key; ensure your key naming scheme keeps related streams on the same shard if you need atomicity.

Actionable Next Steps

1. Prototype: Spin up a local Redis instance and try the example above. Verify that entries persist across restarts by stopping and restarting the container.
2. Measure memory: Use INFO memory to watch used_memory as you add entries. Adjust MAXLEN until the growth stabilizes.
3. Implement a consumer in your language of choice using a client library (e.g., redis-py, ioredis). Use XREADGROUP with BLOCK set to a few milliseconds to balance latency and CPU usage.
4. Set up Redis Sentinel for HA: redis-sentinel /etc/redis/sentinel.conf and configure your application to connect to the sentinel endpoint.
5. Document your consumer group logic: explain how you handle pending entries and how you retry or deduplicate.

By treating Redis Streams as a lightweight, fault‑tolerant log, you can replace heavier message brokers for many use cases while keeping your stack simple.