Diagnosing and Fixing Prometheus 'Out of Order Sample' Errors

The condition

Prometheus rejects a sample and logs out of order sample (or drops the point silently, depending on version) when a data point arrives with a timestamp earlier than the newest sample already stored for that series. The TSDB requires strictly increasing timestamps per series. The useful takeaway: this error is almost never a Prometheus bug — it means two writers disagree about time. Your job is to find which writer is behind.

Common causes and what to check first

Ordered checks

1. Find the offending series. On the Prometheus server, grep the logs: grep -i "out of order" /var/log/syslog (path depends on how you run Prometheus — systemd journal, container logs, or a log file). The log line names the series labels. This tells you whether the problem is one exporter or many.
2. Check NTP on the exporter host. Run chronyc tracking (needs no root) and look at System time and Last offset. If the offset is large or the host is unsynchronized, you have your cause. On VMs, also check the hypervisor host, since guest clock drift often originates there.
3. Check for explicit timestamps in client code. If the exporter is a custom application, search the codebase for timestamp arguments in the client library. Libraries like the Go client allow setting timestamps directly; if the app sets timestamps from a queue event time or batch time, delayed retries will arrive out of order.
4. Check for duplicate writers. If the error appeared after adding a second scrape target, federation, or a second Prometheus server, two writers may be pushing the same series. Look for overlapping scrape configs or a stale target that was supposed to be decommissioned.

Fixes tied to findings

• Clock drift: enable and verify NTP (chronyd, systemd-timesyncd) on the exporter host, then confirm the exporter reports timestamps in the past tense. Do not just restart the exporter — the drift will return.
• Explicit timestamps: remove the timestamp argument from the client library call and let Prometheus assign server time. Trade-off: you lose source event timing, which matters for delayed batch ingestion. If you must keep source timestamps, you need to guarantee monotonicity per series — for example, by serializing writes per series.
• Pushgateway: ensure one pusher per grouping key, or add a unique instance label per pusher so each group of metrics forms its own series.
• Duplicate writers: remove one of the targets. If this is a migration, run the old and new servers against different targets, not the same ones.

The out-of-order window option

Since Prometheus 2.39, the TSDB can accept samples that arrive slightly late. Set --storage.tsdb.out-of-order-time-window (e.g., 10m) on the Prometheus server. This is a grace period, not a fix — use it when delayed samples are legitimate (e.g., batch jobs), not to mask clock drift.

# prometheus --storage.tsdb.out-of-order-time-window=10m

Test in staging first. Increasing the window raises memory overhead and can slow TSDB compaction. If the window is set too high, you may see memory pressure during ingestion spikes.

Escalation criteria

Escalate if: the error persists after NTP is verified; the series labels point to a core infrastructure exporter (node_exporter, kubelet) rather than an app; or the out-of-order samples are coming from a source you don't control (e.g., a third-party exporter). At that point the fix is in the source, not Prometheus.

Verification

After applying a fix, watch the logs for 24 hours. If the error rate drops to zero and the affected series resume ingestion, the cause is confirmed. If the error returns, the fix was incomplete — go back to step 1 and check for a second writer.

Limitations

This guide assumes Prometheus 2.39 or later for the window option; earlier versions reject out-of-order samples with no recourse. The window option is not a substitute for fixing the source. If you cannot fix the source, the window is a workaround, not a solution.