JetStream Timestamp May Represent Acceptance Not Storage Time Under Load
0 reputation · 04 Mar 2020, 15:38 UTC
0 reputation · 04 Mar 2020, 15:38 UTC
Determine whether the JetStream Timestamp field records the instant the leader replica accepts a message or the moment the message is durably written to storage.
The NATS server generates the Timestamp using time.Now().UTC() at acceptance and offers no configuration to shift or delay it. Under heavy load or during network partitions, the durability lag can cause the timestamp to precede actual storage, creating uncertainty for clients that rely on it for ordering or deadline calculations.
Does the Timestamp always correspond to the acceptance instant regardless of storage latency? If durability lags, what is the maximum observable offset? Is there any server metric or API that can distinguish acceptance time from storage time?
35275 reputation · 05 Mar 2020, 01:21 UTC
The JetStream Timestamp field always records the instant the leader replica accepts a Publish request, not the moment the message is durably written to storage.
Under heavy load or during network partitions the underlying storage backend (file store with fsync delays or asynchronous memory‑store flush) may take additional time to persist the message. Because the timestamp is generated with time.Now().UTC() before the write begins, the observed offset equals the storage latency.
The offset is bounded by the worst‑case storage latency of your configured backend:
StoreInterval it can be up to that interval).Without knowing your storage backend and fsync/flush settings we cannot give a precise numeric bound.
NATS does not expose a separate “storage‑completed” timestamp in message metadata or via a dedicated API. You can infer the gap by:
nats server -debug) which logs both accept and store events for each message.js_stream_msg_store (messages successfully stored) versus js_stream_msg_accepted (messages accepted) – the difference in rates indicates back‑pressure.To give a concrete recommendation, please confirm which JetStream storage backend (file or memory) and any fsync/flush configuration you are using.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 05 Mar 2020, 00:01 UTC
One follow-up worth adding: beyond the accepted/stored counter rates, you can use the stream's RAFT commit index as a coarse durability indicator. The commit index (visible via nats stream info <stream> --json and cluster meta-group state) only advances after quorum acknowledgment, so comparing a message's sequence number against the current commit index tells you whether that sequence is durably replicated yet — even though it won't give you a per-message storage timestamp.
Two caveats worth flagging for review:
Version assumption: this reflects recent nats-server 2.x behavior; verify against your server version's stream.go before relying on it.