Redis Streams: A Durable Log Without a Message Broker

You need a durable log, but Kafka feels like too much infrastructure and Redis Pub/Sub loses messages when a consumer disconnects. The gap between those two options is where Redis Streams live: an append-only log with automatic IDs, range queries, and consumer groups that let multiple workers share a topic without double-delivery. Streams have been stable since Redis 5.0, so if you run a recent Redis or a compatible distribution such as Valkey, the commands are likely already available.

The thesis is simple: Streams give you the durability and replay semantics of a lightweight message queue without a separate broker, provided your backlog fits in memory and you accept Redis persistence as your durability boundary.

The problem: Pub/Sub forgets, brokers overwhelm

Redis Pub/Sub is fast, but it is fire-and-forget. If a subscriber is offline, the message is gone. That is fine for ephemeral notifications, but not for order events, audit trails, or job queues where a missed message means lost work.

External brokers solve durability, but they add operational surface: cluster coordination, disk tuning, and separate monitoring. For many teams, that is overkill when the real requirement is a few workers processing a bounded backlog that survives a consumer restart.

Redis Streams close that gap. Entries persist in memory and optionally on disk via AOF or RDB, consumers resume from a stored ID after reconnecting, and consumer groups track per-group offsets so you can scale workers without duplicating work.

How Streams work: an append-only log with offsets

A stream is an ordered sequence of entries. Each entry has an auto-generated ID—typically a millisecond timestamp plus a sequence number—so you can reference, range-query, and replay entries deterministically. The core lifecycle uses a small set of commands:

• XADD appends a new entry and returns its ID.
• XLEN reports the current depth.
• XRANGE / XREVRANGE read a range of entries by ID.
• XREAD reads new entries, optionally blocking with XREAD BLOCK.
• XGROUP CREATE establishes a consumer group with its own offset.
• XREADGROUP reads entries on behalf of a named consumer within a group.
• XACK acknowledges an entry so it leaves the Pending Entries List (PEL).
• XPENDING inspects unacknowledged entries per consumer.

Consumer groups are the key abstraction. Instead of every worker seeing every message, the group tracks a shared cursor. Each worker claims a subset of entries, processes them, and acknowledges. If a worker dies, its unacknowledged entries remain in the Pending Entries List (PEL) and can be reassigned with XCLAIM.

Worked example: producing and consuming from a group

Run the following with redis-cli on a host where you have permission to write keys. If ACLs are enabled, the user needs at least +xadd, +xgroup, +xread, +xreadgroup, and +xack. Replace <stream-key>, <group-name>, and <consumer-name> with your own values.

First, append two events and create a group that starts from the beginning:

XADD <stream-key> * sensor_id 42 temp 19.5
XADD <stream-key> * sensor_id 43 temp 20.1
XGROUP CREATE <stream-key> <group-name> 0

Expected check: XLEN <stream-key> should return 2. The 0 in XGROUP CREATE means the group starts from the first entry; use $ to start only from new entries.

Next, read as a worker and acknowledge:

XREADGROUP GROUP <group-name> <consumer-name> COUNT 10 STREAMS <stream-key> >

Expected check: the reply includes entry IDs and field-value pairs. Copy an entry ID, then acknowledge it:

XACK <stream-key> <group-name> <entry-id>

Expected check: XACK returns the number of entries acknowledged (1 or 0). If you forget to run XACK, the entry stays in the PEL. Verify with:

XPENDING <stream-key> <group-name>

Expected check: the summary shows a count greater than zero, with the oldest pending ID and the consumer that owns it. A growing PEL with a stale oldest ID signals a stuck or crashing worker.

Risk: if you never acknowledge, the PEL grows unbounded and XCLAIM becomes necessary to reassign entries, adding complexity. Treat XACK as part of the same transaction boundary as your business logic. To reset the example locally, DEL <stream-key> removes the stream, its group, and the PEL.

The trade-off: memory-first durability

Streams are memory-first. They live in RAM, and durability depends on Redis persistence and replication. That has two practical consequences.

First, a large backlog increases memory pressure. You can bound a stream with trimming:

XADD <stream-key> MAXLEN 10000 * field value

or trim it later with XTRIM <stream-key> MAXLEN 10000. The tilde variant (MAXLEN ~ 10000) performs approximate trimming, which is cheaper on very large streams because Redis evicts whole macro-nodes rather than checking exact boundaries. The trade-off is that the stream may slightly exceed the specified length.

Second, very large streams or high fan-out can increase fork latency during BGSAVE, because Redis forks the process to write the RDB snapshot. If your stream is hundreds of megabytes, expect a brief latency spike during background saves. Monitor with INFO memory and INFO persistence before and after loading a representative workload.

For observability, XINFO STREAM <stream-key> reports length and last-generated ID, while XINFO GROUPS <stream-key> shows per-group lag. Together with XPENDING, these commands let you detect a stuck consumer before the stream grows unbounded.

What to check before you commit

Before designing an application around Streams, verify your deployment supports them by running these locally or against your target host:

redis-server --version
redis-cli XINFO HELP

If XINFO HELP is unavailable, try COMMAND INFO xadd. Streams have been available since Redis 5.0, but exact syntax and defaults can vary across versions and compatible distributions, so confirm against your actual server.

Start with a bounded stream, a single consumer group, and explicit XACK calls. Measure memory with INFO memory under load. If your backlog fits in RAM, your retention is measured in hours or days rather than weeks, and you do not need complex stream-processing topologies, Redis Streams can replace a dedicated broker with a fraction of the operational cost. If you need multi-day retention, disk-based storage, or Kafka-style partitioning, evaluate a purpose-built log instead.