Whisper timestamp overwrite versus client-side retry tokens for duplicate-free metric ingestion in Carbon

Question

Uncertainty in Carbon Retry Handling
When a client retries a metric write to carbon-cache over TCP after a transport-level failure, it is unclear whether the point has already been durably stored in the Whisper file or merely queued pending acknowledgment. Whisper's design stores a single value per (metric, timestamp) pair and overwrites on replay, which naturally collapses identical retries at storage time.
However, Carbon does not expose an idempotency key or exactly-once token, and a retry may resend the same value after the server has already written it, potentially resulting in a second queued point that survives depending on aggregation rules and flush timing. The uncertainty centers on whether the storage-layer overwrite semantics are sufficient to prevent duplicates, or whether application-level replay control is required.

Does carbon-cache guarantee that a TCP retry of a plaintext metric after successful receipt but before response always results in a single Whisper point at that timestamp?
How do carbon-aggregator rollup rules treat a resent point that arrives after the initial point has been aggregated?
Can a client distinguish a legitimate correction from a redundant retry when only connection errors are observable?

Zoha · Answer

Direct answer
Carbon does not guarantee a single Whisper point for a TCP retry after the server has accepted the data but before the client receives the response; the final Whisper value depends on whether the initial write reached Whisper before the transport failure.
Likely explanation
If the first write arrived in Whisper, the retry overwrites the same (metric, timestamp) slot, leaving one stored point. If the first write was only queued in carbon‑cache or held by carbon‑aggregator, the retry may re‑enter the aggregation path and cause duplicate points.
Confirmed facts

Whisper stores one value per (metric, timestamp) and overwrites later writes to that slot.
Carbon’s plaintext TCP protocol provides no idempotency key, sequence number, or exactly‑once acknowledgment.
carbon‑cache accepts retries after successful receipt but before response, relying on Whisper overwrite semantics.
carbon‑aggregator processes each incoming point independently, so a duplicate timestamp/value pair can be aggregated twice before the rollup window ends.

Steps for this case

Determine whether clients write directly to carbon‑cache or pass through carbon‑aggregator; this decides if rollup duplication is a risk.
If writing directly to carbon‑cache and only a single Whisper point per timestamp is required, send the same metric twice with identical timestamp and verify with a read‑only Whisper inspection.
If using carbon‑aggregator or needing exactly‑once rollup semantics, add client‑side replay control (e.g., embed a unique identifier that your aggregation layer can filter) because Carbon provides no built‑in retry token.
Do not rely on observable connection errors to distinguish a legitimate correction from a redundant retry; a timeout tells you nothing about persistence.

Verification
Run read‑only commands on a non‑production Carbon instance:
whisper-info.py /var/lib/carbon/whisper/example.metric.wsp
whisper-dump.py /var/lib/carbon/whisper/example.metric.wsp
Then send the same metric twice with the same timestamp, compare the Whisper output, and check carbon‑aggregator rollup output if applicable. Results from your instance are required before treating any behavior as confirmed.
Missing diagnostic detail
Are your clients sending metrics directly to carbon‑cache, or through carbon‑aggregator? This determines whether you need rollup deduplication in addition to Whisper overwrite semantics.

Whisper timestamp overwrite versus client-side retry tokens for duplicate-free metric ingestion in Carbon

Uncertainty in Carbon Retry Handling

1 answer

Direct answer

Likely explanation

Confirmed facts

Steps for this case

Verification

Missing diagnostic detail

Your perspective belongs here.

0 question comments