Go HTTP Service: Diagnosing Goroutine Leaks That Outlive Requests

Recognize the condition

A Go HTTP service that restarts cleanly but gradually consumes more memory over days is often leaking goroutines. A goroutine is a lightweight execution unit managed by the Go runtime. When a handler starts a goroutine and that goroutine never exits—because it is blocked on a channel send, channel receive, or mutex—the runtime keeps its stack allocated. After the HTTP response is sent, those goroutines remain alive.

Watch for:

• runtime.NumGoroutine() does not return to baseline after traffic stops
• steady memory growth with a flat request rate
• handler latency rises while CPU stays flat
• out-of-memory kills followed by temporary recovery after a restart

Common causes and matching signals

Observed behavior	Likely cause	Where it appears in a goroutine profile
Handler returns but goroutine count stays high	Goroutine started in the handler waits on an unbuffered channel with no active receiver	Stack shows chan send
Worker loops survive service shutdown	Receive from a channel that is never closed, with no cancellation signal	Stack shows chan receive inside a select loop
Count climbs mainly during concurrent writes	Mutex locked on an error path without a matching unlock	Stack shows mutex lock
Producers block only after sustained burst	Buffered channel fills faster than consumers drain	Stack shows chan send on a full buffer

Ordered checks

1. Establish a baseline with runtime.NumGoroutine()

runtime.NumGoroutine() returns the number of goroutines currently alive. Call it from a metrics endpoint or a test while the application is idle, then again after sustained traffic.

import "runtime"

func currentGoroutines() int {
    return runtime.NumGoroutine()
}

Where to run: inside your application code or a Go test. Required permissions: none. Expected check: the count stays flat over several minutes of steady traffic. Risk: none; this is a read-only call.

2. Expose the pprof goroutine endpoint

pprof is Go's built-in profiling toolkit. Importing net/http/pprof registers debugging endpoints, including one that lists every goroutine and its stack trace.

import (
    _ "net/http/pprof"
    "net/http"
)

func main() {
    go func() {
        err := http.ListenAndServe("localhost:6060", nil)
        if err != nil {
            return
        }
    }()
    // application code
}

Where to run: on a host that can reach the running instance. Required permissions: network access to the debug port. Expected check: curl http://localhost:6060/debug/pprof/goroutine?debug=1 returns stack traces for all active goroutines. Risk: the endpoint exposes internal program state; do not expose it publicly without authentication or a firewall.

3. Capture an idle profile and a loaded profile

curl http://localhost:6060/debug/pprof/goroutine?debug=1 > goroutine-idle.txt
# run load test, then
curl http://localhost:6060/debug/pprof/goroutine?debug=1 > goroutine-load.txt

Where to run: a terminal with access to the service. Required permissions: none beyond network access. Expected check: the loaded profile has more entries than idle. If the count does not return close to baseline after the load stops, you likely have a leak.

4. Read the stack traces

Look for goroutines stuck in chan send, chan receive, or mutex lock. The file and line number in each stack points to the blocking operation, not necessarily the line that created the goroutine.

Fixes tied to the finding

Blocked send: no receiver is available

If a sender blocks on an unbuffered channel, add a timeout or a fallback so the sender cannot wait forever.

select {
case ch <- value:
    // sent
case <-ctx.Done():
    // caller canceled; stop trying
case <-time.After(5 * time.Second):
    // give up after a bounded wait
}

Blocked receive: channel is never closed

Close a channel when no more values will be sent, or use context cancellation so receivers can exit.

for {
    select {
    case <-ctx.Done():
        return
    case value, ok := <-jobs:
        if !ok {
            return // channel closed
        }
        process(value)
    }
}

Mutex held across an error path

Call mu.Unlock() with defer immediately after locking, so an early return cannot leave the mutex locked.

mu.Lock()
defer mu.Unlock()

Concrete example: request-scoped worker

The following structure shows the pattern. It is not a tested production snippet.

func handler(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    jobs := make(chan int, 5)
    results := make(chan int, 5)
    var wg sync.WaitGroup

    wg.Add(1)
    go func() {
        defer wg.Done()
        for {
            select {
            case <-ctx.Done():
                return
            case job, ok := <-jobs:
                if !ok {
                    return
                }
                results <- job * 2
            }
        }
    }()

    // send work, then close(jobs)
    // wg.Wait() before writing the response
}

The ctx.Done() case lets the worker exit when the HTTP request is canceled or times out. The ok == false check lets it exit when the job channel is closed. Without either, the worker can block forever on <-jobs even after the handler has returned.

Limitations

• runtime.NumGoroutine() can spike legitimately during traffic bursts; alert on a sustained trend, not a single sample.
• pprof adds CPU overhead. Enable it temporarily for diagnosis, or gate it behind authentication in production.
• A profile shows where a goroutine is blocked, not why the application logic sent it there. You still need to trace the caller.

Practical verification

1. Write a test that records runtime.NumGoroutine() before and after the suspect handler. The count should return to baseline once the handler completes and its goroutines exit.
2. Run a load test against the fixed build. Memory should plateau rather than grow linearly with request count.
3. Reload /debug/pprof/goroutine?debug=1 after the fix. Stacks that previously showed chan receive should no longer accumulate.

Escalation criteria

• Goroutine count keeps rising after you close channels and add context cancellation.
• The blocking stack is inside a dependency or standard library call you do not control.
• You cannot safely enable pprof in the production environment where the leak appears.

Because the fixes are code changes, rollback is a standard deploy revert; these diagnostics do not change database or persistent state.