Persistent queue recovery verification limits in OpenTelemetry Collector

Direct Answer

The Collector does not provide a built-in checksum, manifest, or reconciliation report for the file_storage-backed persistent queue. Completeness cannot be verified internally without external tooling; it must be inferred from Collector metrics and exporter telemetry. There are no documented hard limits on duplicate reconciliation; the practical limit is backend ingest capacity and Collector disk and memory during replay.

Confirmed facts

• The Collector implements at-least-once delivery for the persistent queue. No deduplication is performed by the Collector.
• No internal checksum or manifest is exposed for recovered queue data.
• Queue drain is indicated by otelcol_extension_file_storage_queue_size gauge reaching zero and staying zero longer than the export timeout.
• Exporter health is indicated by otelcol_exporter_send_success_spans and otelcol_exporter_send_failed_spans, or the equivalent metric for metrics and logs.

Likely explanation

Because the queue is file-backed, the Collector replays items after restart until exporter send succeeds. Duplicate telemetry can occur if acknowledgment was received before an unclean shutdown but not persisted in the Collector's in-memory state. Gaps are unlikely unless disk fills or the queue file is corrupted, but the Collector does not report gaps explicitly.

Verification steps for this case

1. After restart, watch otelcol_extension_file_storage_queue_size. Drain is likely complete when it is 0 for a period longer than the exporter timeout.
2. Confirm otelcol_exporter_send_success_spans or the equivalent metric for metrics and logs increases while otelcol_exporter_send_failed_spans stays at zero during the drain window.
3. Optionally enable debug logging for file_storage and look for log entries indicating queue flush.

Practical caveats

• Queue size alone can miss in-flight batches; cross-check exporter send counters.
• Without backend-side deduplication, unclean shutdowns produce observable duplicate telemetry.
• file_storage does not auto-rotate the queue file; monitor disk usage and queue file size during replay.

One missing diagnostic detail that changes the recommendation: which exporter is in use. Metric names for send success and failure vary by exporter, and some backends expose deduplication signals that strengthen verification.