Hunting Deadlocks in Go: Using GoLand and Delve for Goroutine Inspection

The Invisible Wall: When Your Go App Just Stops

\n

You have a Go service that works perfectly in unit tests but occasionally hangs in production or during integration tests. There are no panic logs, no stack traces in the output, and the CPU usage has dropped to near zero. You are likely facing a deadlock—a state where two or more goroutines are waiting for each other to release resources, and none can proceed.

\n\n

Visualizing the Goroutine Stack

\n

When a Go program hangs, the most critical piece of information is not the current line of code, but the state of all other goroutines. GoLand's debugger provides a Goroutines view that lists every active goroutine, its current status (Running, Waiting, or Sleeping), and its current stack trace.

\n\n

By pausing a hanging application, you can scan this list to find goroutines stuck on channel operations. If you see multiple goroutines waiting on chan receive or chan send without a corresponding sender or receiver, you have found your deadlock site. This is significantly faster than manually analyzing a pprof dump because you can click a goroutine in the list and be taken directly to the line of code causing the block.

\n\n

Isolating Noise with Conditional Breakpoints

\n

Concurrency bugs often only trigger after thousands of successful iterations. Stopping the program at every loop iteration is impractical and destroys the timing of the execution, making the bug disappear (a “Heisenbug”).

\n\n

Conditional breakpoints allow you to tell the debugger: “Only stop here if i == 5000” or “Stop here if userID == \"test-123\". This allows the application to run at near-native speed until the specific edge case is hit, preserving the state of the system as closely as possible to the failure condition.

\n\n

Worked Example: Diagnosing a Circular Wait

\n

Consider a scenario where two goroutines are trying to lock two different mutexes in opposite orders. This is a classic circular dependency.

\n\n

// Run this with: go run -gcflags=\\"all=-N -l\\" main.go
package main

import (
    "fmt"
    "sync"
    "time"
)

var lockA = &sync.Mutex{}
var lockB = &sync.Mutex{}

func main() {
    go func() {
        lockA.Lock()
        fmt.Println(\\"Goroutine 1: Locked A\\")
        time.Sleep(time.Millisecond * 10)
        lockB.Lock() // Will hang here
    }()

    go func() {
        lockB.Lock()
        fmt.Println(\\"Goroutine 2: Locked B\\")
        time.Sleep(time.Millisecond * 10)
        lockA.Lock() // Will hang here
    }()

    select {}
}

\n\n

Diagnostic Steps in GoLand

\n

\n
1. Build Configuration: Ensure you are running with -gcflags=\\"all=-N -l\\". This disables compiler optimizations and inlining, ensuring that the debugger can see all local variables and accurate line numbers.
\n
2. Trigger the Hang: Run the program in Debug mode. The console will show the first two print statements, then stop.
\n
3. Pause Execution: Click the Pause Program button in the Debug tool window.
\n
4. Inspect Goroutines: Open the Goroutines tab. You will see two goroutines blocked on sync.runtime_SemacquireMutex.
\n
5. Trace the Chain: Clicking the first goroutine shows it is waiting for lockB while holding lockA. Clicking the second shows it is waiting for lockA while holding lockB. The circular dependency is now visually confirmed.
\n

\n\n

Trade-offs and Debugging Risks

\n

While powerful, using a debugger for concurrency introduces specific risks:

\n

\n
• Timing Shifts: Pausing one goroutine does not always pause the entire world instantly. The act of hitting a breakpoint can change the interleaving of threads, potentially masking the race condition you are trying to find.
\n
• Performance Overhead: Running with -N -l flags makes the binary slower. In extremely high-throughput systems, the debugger might introduce enough latency to prevent the deadlock from occurring.
\n
• State Mutation: Be cautious when using the “Evaluate Expression” tool to change variable values while paused; modifying a mutex state manually can lead to unpredictable crashes.
\n

\n\n

Verification and Closing

\n

To verify your fix, restart the session without the pause button and monitor the Goroutines view periodically. If the number of goroutines remains stable and the application continues to process requests, the deadlock is resolved. For long-running services, use the Attach to Process feature in GoLand to inspect a live instance without restarting the entire environment, provided the process was started with the necessary debug symbols.