Diagnosing ASP.NET Core Thread‑Pool Starvation from Sync‑Over‑Async Blocking

Recognizable Condition

When a web application’s request latency climbs in discrete steps—p99 jumps from 200 ms to 800 ms to 2 s—while CPU usage stays around 10–30 %, you’re likely seeing thread‑pool starvation. The symptoms are:

• Steady CPU, low memory pressure.
• ThreadPool queue length grows to hundreds or thousands.
• Thread count climbs slowly toward the configured maximum.
• Requests time out or return 503 under load.
• All endpoints affected, not just a single route.

These patterns often surface after adding code that blocks an async I/O operation, such as .Result, .Wait, or Task.Run().Result inside a request pipeline.

Cause & Diagnostic Table

Signal	Likely Cause
Stack contains Wait or Result	Sync‑over‑async blocking
ThreadPool queue length high, completed‑items rate low	ThreadPool min threads too low or long ramp‑up
Connection resets or 503s after heavy load	Kestrel backlog or connection limit reached
Common stack frames across all requests	Blocking in middleware/filters

Ordered Checks

1. Capture EventCounters

   Run the counters against the running process. Replace 12345 with your process ID.

   dotnet-counters monitor --process-id 12345 \
     --counters System.Threading.ThreadPool,System.Runtime \
     --period 1
   

   Look for ThreadPoolQueueLength and ThreadPoolThreadCount. A steady rise without a corresponding CPU spike confirms starvation.

2. Collect a Full Dump at Peak Latency

   Use dotnet-dump to grab a snapshot. This requires the user to have read access to the process.

   dotnet-dump collect --process-id 12345 --dump-file dump.dmp
   

   Store the file in a safe location; it can be large (hundreds of MB).

3. Analyze Stack Traces

   Open the dump with dotnet-dump analyze or a tool like WinDbg. Search for frames containing Wait, Result, Thread.Sleep, or synchronous I/O APIs (e.g., FileStream.Read, SqlConnection.Open).

   Example snippet (pseudo‑output):

   0x00007ff...  MyApp.Controllers.HomeController.Index (Async) 
   0x00007ff...  System.Threading.Tasks.Task.get_Result() 
   0x00007ff...  System.Net.Http.HttpClient.GetAsync() 
   

4. Review Recent Code Changes

   Check pull requests or commits that added .Result, .Wait, or wrapped async work in Task.Run().Result. Also look for Thread.Sleep inside request‑handling code.

5. Validate ThreadPool and Kestrel Limits

   By default, .NET Core sets the thread‑pool min threads to 0, allowing a slow ramp‑up. If you see a queue that grows but the thread count stays near the minimum, consider ThreadPool.SetMinThreads as a temporary mitigation. Kestrel’s default backlog is 100; increase ListenOptions.Backlog only after confirming that the backlog is the real bottleneck.

   public void Configure(KestrelServerOptions options) {
       options.Listen(IPAddress.Any, 5000, listenOptions => {
           listenOptions.Backlog = 200; // temporary
       });
   }
   

Fixes Tied to Findings

• Convert sync‑over‑async to true async

  Replace .Result or .Wait with await throughout the request pipeline. Ensure all called methods are fully async.

  // Bad
  var data = dbContext.Users.FirstOrDefaultAsync(u => u.Id == id).Result;
  
  // Good
  var data = await dbContext.Users.FirstOrDefaultAsync(u => u.Id == id);
  

• Remove blocking calls in middleware

  Middleware that calls Thread.Sleep or synchronous I/O should be rewritten to async equivalents or moved out of the request path.

• Reserve Task.Run for CPU‑bound work only

  Do not use Task.Run().Result to offload I/O; instead, use the async API directly.

• Temporary thread‑pool mitigation

  If you must keep a sync call for a short period, raise the min threads:

  ThreadPool.SetMinThreads(200, 200); // 200 worker + 200 I/O threads
  

  Remember: this hides the real issue and can increase context switching.

• ConfigureAwait(false) in libraries

  Use ConfigureAwait(false) in library code that does not depend on the ASP.NET synchronization context. Avoid it in request handlers that need HttpContext or services resolved from the request scope.

• Use ValueTask only after profiling

  Hot paths that return a value synchronously can use ValueTask to reduce allocations, but only if you’ve measured the benefit.

Escalation Criteria

1. If, after removing all blocking calls, the queue length still climbs and latency remains high, suspect a runtime bug or a GC pause. Capture a GC histogram with dotnet-trace collect --providers Microsoft-Windows-DotNETRuntime:0x0000000000000400 and analyze for long GC pauses.
2. If the problem persists on a clean install of the same .NET version, file a bug with Microsoft. Include:
   • Runtime version and OS details.
   • EventCounter export (CSV).
   • Full dump and stack analysis.
   • A minimal reproducible repro that triggers the stall.
3. Do not raise ThreadPool.SetMinThreads permanently; treat it as a mitigation until the async refactor is complete.

Practical Validation Checklist

• Run a staged load test with the same traffic profile that produced the stall.
• Verify that dotnet-counters shows ThreadPoolQueueLength staying below 50 and ThreadPoolThreadCount near the default minimum.
• Assert that p99 latency remains under the SLA threshold (e.g., <200 ms).
• Confirm that the dump no longer contains Wait or Result frames.

Limitations and Caveats

• EventCounter names and default values vary between .NET 6, 7, 8, and hosting models (IIS, Docker, Kestrel). Verify against the exact runtime you’re troubleshooting.
• Collecting dumps and traces adds overhead; perform them during controlled testing, not during an unmitigated production incident.
• Raising thread‑pool min threads can mask the real issue and increase memory usage; use only as a temporary stopgap.
• ConfigureAwait(false) can break code that relies on the request synchronization context; apply it only in library code that does not touch HttpContext or scoped services.

Conclusion

Thread‑pool starvation from sync‑over‑async blocking is a common cause of sudden ASP.NET Core latency spikes. By following the ordered checks above—capturing counters, dumping the process, analyzing stacks, reviewing recent changes, and validating thread‑pool settings—you can pinpoint the offending code. Replace blocking calls with true async, only use ThreadPool.SetMinThreads as a temporary mitigation, and escalated to runtime owners only when the root cause remains elusive.