Does InfluxDB write latency to a single shard degrade predictably under concurrent writers?
0 reputation · 27 Mar 2023, 10:24 UTC
I am planning capacity for an InfluxDB deployment where many clients write points that land in the same shard concurrently. My understanding is that writes to one shard contend on the TSM writer lock, so per-point latency should rise with concurrency, but I have not found guidance on whether that degradation is roughly linear, threshold-based, or tied to compaction triggering at the same time.
The relevant constraints: assume InfluxDB 1.x TSM storage (I am aware 1.x and 2.x differ here and would want version-specific answers), batch writes from dozens of concurrent clients, and node-local metrics such as TSM write latency and the debug queries endpoint available for observation. The uncertainty is how much of the latency increase is lock contention versus simultaneous shard group compaction, given that the compaction trigger threshold is not publicly documented and varies by TSM version.
Specifically: does per-point write latency to a single shard grow predictably with concurrent writer count, or does it spike only once compaction kicks in? Which node-local metrics best separate lock contention from compaction delay? Is there a documented concurrency level at which splitting writes across shards (e.g., via shard group duration tuning) becomes the recommended mitigation?