Does InfluxDB write latency to a single shard degrade predictably under concurrent writers?

Short answer

For InfluxDB 1.x TSM, per-point write latency to a single shard degrades gradually with concurrent writer count due to per-shard lock contention, and then spikes whenever a cache snapshot/flush or a TSM compaction in the same shard group runs. It is neither purely linear nor purely threshold-based – expect a rising baseline with periodic spikes, i.e., a bimodal latency distribution.

Likely explanation vs. confirmed facts

What is well established for 1.x TSM:

• Concurrent writes are accepted into the WAL and in-memory cache in parallel, but cache snapshotting and flush to TSM files serialize per shard. Dozens of writers therefore queue, and latency climbs moderately rather than staying flat.
• When the cache crosses its size or age thresholds (cache-max-memory-size, cache-snapshot-memory-size), a forced snapshot/flush produces a pronounced latency spike.
• Level compactions rewrite TSM files in the same shard group, consuming disk I/O and briefly blocking work on affected files — a second, independent source of spikes.

What is not established: any fixed concurrency number at which degradation becomes unacceptable, and the exact compaction trigger thresholds. Both vary by 1.x minor version, config, and hardware, so treat any specific threshold you see quoted as workload-dependent, not a documented constant.

Separating lock contention from compaction delay

The two causes have different signatures in node-local metrics:

• Cache flush signature: latency spikes that coincide with rising snapshotCount and resets of cacheAgeMs in the per-shard tsm1 stats.
• Compaction signature: spikes aligned with compaction queue depth / compaction counts per level.
• Pure contention signature: p50/p99 latency rising smoothly with writer count between those events.

Poll /debug/vars every few seconds during a load test and log the tsm1 per-shard cache and compaction counters alongside client-side latency percentiles. Correlating spike timestamps with snapshots versus compaction activity attributes the delay. The _internal monitor database exposes similar counters, but note it adds write load itself and can slightly worsen contention on a saturated node — prefer /debug/vars for measurement.

Verifying the curve for your hardware

1. On a test instance, fix total point throughput and batch size; vary concurrent clients (e.g., 1, 8, 32, 64).
2. Record p50/p99 write latency per concurrency level, plus the /debug/vars counters above.
3. Check whether spikes coincide with snapshots/compactions or rise smoothly – that tells you which mitigation applies.

Mitigations, in order of effort

Batch size first. Larger batches amortize lock acquisition and WAL fsync; dozens of clients sending tiny batches degrade faster than the same throughput in fewer, larger batches. This often flattens the curve without any schema change.

Shard group duration second. Shortening it (e.g., 7d → 1d, or hours for very high throughput) spreads concurrent writes across more shards and makes compactions touch smaller files. There is no documented concurrency level at which this becomes the recommended step — it is the standard mitigation once batching is maxed out and p99 is still dominated by single-shard contention. The trade-off is more shards and files to manage, and it only helps if your writes actually span the new shard boundaries.

Validate whichever change you pick by re-running the same load test and confirming p99 flattens before applying it in production.

Caveats

All of this is specific to InfluxDB 1.x TSM. 2.x and especially 3.x use different storage engines, so do not extrapolate. Exact compaction thresholds and queue behavior vary across 1.x minor versions and config (compact-throughput, cache sizes) — verify against your version's release notes before tuning.