GC stop-the-world pauses and request latency under concurrent load in Go

Latency only visible under concurrent requests

I am designing a Go service where p99 request latency must stay low even when many goroutines handle requests simultaneously. My concern is the garbage collector: under high concurrency the allocation rate rises, GC cycles become more frequent, and stop-the-world phases such as mark termination and sweep termination can delay otherwise fast requests. I have not yet measured this in production, so I am treating it as a design risk rather than a confirmed incident.

Constraints and uncertainty

The service targets a recent Go release (1.14 or newer, so asynchronous preemption applies), and I cannot easily reduce GOMAXPROCS because throughput matters. I understand the GC pacer adjusts collection frequency based on heap growth, which makes pause timing hard to predict when allocation patterns are bursty. I also know STW phases can scale with the number of active processors, but I am unsure how much that matters in practice versus allocation rate.

I plan to verify with go tool trace and GODEBUG=gctrace=1 under a load test, but I want to reason about the design first.

Questions

• Which GC phase is most likely to dominate tail latency under concurrent load: mark termination, sweep termination, or write-barrier overhead during concurrent marking?
• Does lowering allocation rate per request reliably reduce STW frequency, or does the pacer still trigger cycles at similar intervals?
• Are there documented runtime metrics that distinguish scheduler latency from GC pause time when both appear under concurrency?