Choosing the Right Start ID for Redis Streams Consumer Groups

Problem: Deciding where a consumer group should begin reading

When you create a Redis Streams consumer group you must pick a start ID. The choice determines whether the group processes only new messages, replays the entire stream, or begins at a specific point. Picking the wrong start ID can cause you to miss important data, replay old work unnecessarily, or create unexpected pending‑entry buildup.

Thesis: The start ID is a simple but consequential engineering decision – use '$' for live‑only processing and '0' for full back‑fill, and verify the choice with XPENDING and XLEN.

Understanding the start ID options

Redis Streams consumer groups are created with XGROUP CREATE. The third argument is the start ID:

• $ – the group begins with messages that arrive after the command runs. Existing entries in the stream are ignored.
• 0 – the group starts at the very first entry in the stream, allowing replay of all historic data.
• A specific entry ID (e.g., 1697045600000-0) – start at that exact point.

The start ID does not affect how pending entries are tracked; it only sets the initial read position for the group.

Worked example: Comparing '$' and '0' start IDs

1. Start a Redis server (≥5.0) and open two CLI sessions.
2. Create a stream and add a few historic messages:

XADD mystream * sensor temperature 22.0
XADD mystream * sensor temperature 22.5
XADD mystream * sensor temperature 23.0

1. Create a consumer group that will only see new messages (start ID '$'):

XGROUP CREATE mystream livegroup $

1. Create a second group that will replay the whole stream (start ID '0'):

XGROUP CREATE mystream backfillgroup 0

1. Add a new message while both groups exist:

XADD mystream * sensor temperature 23.2

1. Read from each group using a unique consumer name:

XREADGROUP GROUP livegroup consumerA COUNT 1 STREAMS mystream >
XREADGROUP GROUP backfillgroup consumerB COUNT 1 STREAMS mystream >

The first command returns the newly added message (ID greater than the historic entries). The second command returns the oldest unacknowledged message from the stream (the first historic entry). This demonstrates how the start ID controls the initial data set each group sees.

1. Check pending entries for each group to confirm the server is tracking them correctly:

XPENDING mystream livegroup
XPENDING mystream backfillgroup

Each group shows one pending entry corresponding to the message it just delivered.

Trade‑offs and operational tips

• Replay vs. live processing: Using '0' is useful for back‑filling a new consumer or repairing gaps, but it can increase processing load and cause duplicate work if consumers have already handled those messages. Using '$' avoids reprocessing history but risks missing data that arrived before the group was created.
• Pending‑entry growth: Regardless of start ID, any message delivered but not acknowledged stays in the pending list. Monitor with XPENDING and set alerts if the count exceeds a threshold (e.g., 10 000 entries) to prevent memory pressure.
• Changing the start ID: The start ID of an existing group cannot be altered. To change behavior you must create a new group with a different start ID and, if needed, migrate consumer state.

Actionable closing: How to pick and verify

1. Determine whether you need historic replay. If yes, create the group with start ID '0' (or a specific historic ID). If you only care about incoming data, use '$'. 2. After creating the group, add a test message and run XREADGROUP for each consumer to confirm you receive the expected set of entries. 3. Use XPENDING to verify that the server is tracking delivered messages and that acknowledgments (XACK) reduce the pending count. 4. Set a routine check (e.g., via a cron job or monitoring script) that runs XPENDING and XLEN mystream to ensure pending entries stay within acceptable bounds. By matching the start ID to your processing goal and validating with the simple commands above, you can avoid silent data loss or unnecessary replay while keeping the consumer group’s overhead predictable.