NATS JetStream Stream ↔ Consumer: Stale Message Retrieval with max_age Retention

In NATS JetStream, consumers can still retrieve messages that have technically exceeded their max_age limit. This occurs because JetStream does not delete messages the instant they expire; instead, it marks them for removal and purges them during periodic housekeeping cycles.

The Role of Housekeeping

The max_age retention policy defines the maximum time a message can stay in a stream. Once a message passes this age, it becomes eligible for deletion. However, the actual removal is performed by the stream leader during a background task. In recent NATS releases, this cycle occurs approximately every 30 seconds by default. Until this cycle completes, the message remains in the store and is visible to all active consumers.

Clustered Deployments and Replicas

In a clustered environment, the situation is further complicated by replication lag. While the leader manages the housekeeping, replicas must synchronize these state changes. If a consumer is connected to a replica that has not yet processed the leader's purge command, the consumer may still see and fetch the expired message. This creates a nondeterministic window of visibility that extends beyond the leader's own housekeeping interval.

Reliability and Implementation

Consumers cannot rely on the JetStream API to guarantee that a message has been purged based on max_age. If your application requires strict adherence to message age, you must implement the following strategies:

• Client-Side Validation: Include a timestamp in the message payload and have the consumer discard the message if the current time exceeds the allowed age.
• Filtering: Use consumer filters where possible to exclude older data, though this does not solve the underlying issue of the message existing in the store.
• Tuning Housekeeping: While the housekeeping interval can be tuned via server configuration, it is generally intended to optimize performance rather than as a mechanism for real-time deletion.

To verify the specific window in your environment, you can monitor a stream with a short max_age and observe the time between the expiration timestamp and the moment the message returns a 404 Not Found or disappears from a pull query.