Indexing Retried Events: Avoiding Duplicate Writes in Splunk Forwarders

Question

Problem Statement
The goal is to guarantee that events sent from a Splunk forwarder are written to an index only once, even when transient network failures trigger retries. The forwarder can be configured with indexer_ack = true and a retry stanza, while the indexer may reject duplicates based on event IDs. However, it is unclear whether this combination provides true exactly‑once delivery or if duplicates can still slip through.
Key Constraints

indexer_ack forces the forwarder to wait for acknowledgement before clearing its buffer.
The retry stanza controls the number of resend attempts and back‑off strategy.
Indexers can reject duplicates using duplicate_check, but the interaction with event IDs is version‑dependent.
The dedup search command removes duplicates at search time, not ingestion time.

Unresolved Questions

Does enabling indexer_ack together with the default retry policy guarantee that a retried event is not written twice, or can duplicates still appear if the indexer acknowledges before dedup logic runs?
What is the current behavior of the indexer’s duplicate_check setting when indexer_ack is true, and how does it interact with the event’s event_id field across recent Splunk Enterprise releases?
Should administrators rely on search‑time dedup or configure indexer‑level duplicate detection to achieve exactly‑once semantics, and what trade‑offs exist between the two approaches?

Tasadduq Burney · Answer

Short Answer
Enabling indexer_ack = true and the retry stanza only guarantees that the forwarder keeps an event in its queue until an explicit ACK is received.  It does not by itself prevent a duplicate from being written if the indexer has already ACKed the event and the forwarder retries after a timeout or restart.  To achieve true exactly‑once delivery you must also enable duplicate_check = true on the indexer and ensure every forwarded event carries a stable, unique event_id (or _guid) field.  Without that field the indexer cannot recognize a retry as a duplicate, and a second copy will be indexed.
Why the Combination Doesn’t Suffice Alone

indexer_ack only controls the forwarder’s local buffer clearing.
The retry stanza dictates how many times the forwarder will re‑send an event after a failure, but it does not check whether the event was already written.
Duplicate detection happens on the indexer side; if duplicate_check is off or the event lacks a unique key, the indexer will happily accept the resend.
Search‑time dedup merely hides duplicates in query results and does not affect the indexed data volume.

Current Behaviour of duplicate_check
In Splunk Enterprise 7.x and 8.x the indexer calculates a hash of the event’s event_id (or _guid if present) before it writes the event to disk.  If the hash matches an existing entry, the event is discarded and a duplicate_check drop is logged.  The exact field used and the timing of the check (before or after the ACK is recorded) can vary between major releases, so it is essential to confirm the behaviour on your specific version.
Recommended Steps for Exactly‑Once Delivery

Verify Indexer Settings

Check duplicate_check = true in the indexer’s indexes.conf for the target index.
Confirm the indexer is running a supported major version (7.x or 8.x) where event_id based dedup is documented.

Ensure a Stable Event ID

Configure the forwarder or data source to populate a field named event_id (or _guid) with a deterministic value for each event.
Validate that the field is not altered downstream (e.g., by transforms or props.conf).

Test the Flow

Generate a single event with a known event_id and send it with indexer_ack = true.
Simulate a network hiccup (e.g., block port 8089) to force a retry.
Check the forwarder’s metrics.log for a retry counter and the indexer’s splunkd.log for a duplicate_check drop message.

Monitor & Tune

Use | metrics | search type=forwarder | stats avg(queue_size) by index to observe queue behaviour during retries.
Adjust the retry stanza’s max_attempts or back‑off settings if you see excessive retries without dedup drops.

What to Ask Next
To refine this configuration, could you confirm:

Which Splunk Enterprise major version your indexer is running?
Whether the target index already has duplicate_check = true enabled?
Which field is currently being used as the event’s unique identifier (event_id, _guid, or a custom field)?

Answering these will help us determine if any additional tuning is required.

Indexing Retried Events: Avoiding Duplicate Writes in Splunk Forwarders

Problem Statement

Key Constraints

Unresolved Questions

1 answer

Short Answer

Why the Combination Doesn’t Suffice Alone

Current Behaviour of `duplicate_check`

Recommended Steps for Exactly‑Once Delivery

What to Ask Next

Your perspective belongs here.

0 question comments