GC stop-the-world pauses and request latency under concurrent load in Go

Question

Latency only visible under concurrent requests

I am designing a Go service where p99 request latency must stay low even when many goroutines handle requests simultaneously. My concern is the garbage collector: under high concurrency the allocation rate rises, GC cycles become more frequent, and stop-the-world phases such as mark termination and sweep termination can delay otherwise fast requests. I have not yet measured this in production, so I am treating it as a design risk rather than a confirmed incident.

Constraints and uncertainty

The service targets a recent Go release (1.14 or newer, so asynchronous preemption applies), and I cannot easily reduce GOMAXPROCS because throughput matters. I understand the GC pacer adjusts collection frequency based on heap growth, which makes pause timing hard to predict when allocation patterns are bursty. I also know STW phases can scale with the number of active processors, but I am unsure how much that matters in practice versus allocation rate.

I plan to verify with go tool trace and GODEBUG=gctrace=1 under a load test, but I want to reason about the design first.

Questions

Which GC phase is most likely to dominate tail latency under concurrent load: mark termination, sweep termination, or write-barrier overhead during concurrent marking?
Does lowering allocation rate per request reliably reduce STW frequency, or does the pacer still trigger cycles at similar intervals?
Are there documented runtime metrics that distinguish scheduler latency from GC pause time when both appear under concurrency?

Tasadduq Burney · Answer

Answer
Under concurrent load the mark‑termination stop‑the‑world (STW) phase is usually the biggest contributor to tail latency.
Confirmed facts

Mark‑termination STW blocks all processors while scanning remaining gray objects; its duration grows with live heap size and number of active goroutines.
Sweep‑termination STW is typically much shorter because it runs concurrently with the mutator after mark‑termination and only touches already‑marked spans.
Write‑barrier overhead adds CPU cost but does not cause STW pauses; it executes alongside mutator work.

Likely explanation
When many goroutines allocate simultaneously, the allocation rate rises, prompting the GC pacer to start a cycle sooner. The concurrent mark phase proceeds with write‑barrier assistance, but once the gray worklist is drained the runtime must enter mark‑termination STW to finish the scan. Because all Ps are stopped, any request in flight experiences a pause whose length correlates with the heap size at that moment, producing the observed p99 spikes.
Verification steps

Run a load test (e.g., hey -z 30s -c 100 http://localhost:8080/) with GODEBUG=gctrace=1 enabled; note the timestamps and durations of each STW line.
Generate a trace: go tool trace -o trace.out ./mybin under the same load, then open with go tool trace and look for intervals where all Ps are in _Goff (GC STW) versus intervals where individual Ps are in _Gwaiting or _Grunnable (scheduler latency).
Optionally, poll runtime.ReadMemStats at request boundaries to accumulate PauseNs and compare with latency spikes from the trace.

GC stop-the-world pauses and request latency under concurrent load in Go

Latency only visible under concurrent requests

Constraints and uncertainty

Questions

1 answer

Answer

Confirmed facts

Likely explanation

Verification steps

Your perspective belongs here.

0 question comments