Diagnosing Thread Pool Starvation in ASP.NET Core Applications

Recognizable condition

Under load, request latency rises steadily while CPU utilization stays low. The ThreadPool reports a dwindling number of available worker threads and a growing queue of pending work items. When the load eases, latency drops in bursts because the pool adds new threads slowly (about one per 500 ms interval) rather than instantly.

Cause / diagnostic table

Symptom	Likely cause	What to look for
Latency ↑, CPU low, worker threads ↓	Sync‑over‑async blocking (e.g., .Result, Task.Wait())	Threads stuck in WaitOne, Monitor.Enter, or Task.InternalWait
Same symptom, but stacks show long CPU work	CPU‑bound work queued to the pool without Task.Run	Threads executing heavy computation inside request handlers
Latency spikes with high lock contention	Excessive locking or monitor contention	Many threads in Monitor.Enter or SpinWait loops
Background timers or I/O compete	Timer callbacks or non‑request work sharing the pool	Threads in TimerCallback or ThreadPool.QueueUserWorkItem

Ordered checks

1. Capture runtime metrics

   While reproducing the load, run:

   dotnet-counters monitor --process-id <pid> \
      ThreadPool-ThreadCount,ThreadPool-QueueLength,ThreadPool-CompletedWorkItemCount

   Required permission: ability to attach to the process (usually the same user that runs the app). Risk: minimal; counters are read‑only.

   Look for ThreadPool‑ThreadCount staying near the min worker count while ThreadPool‑QueueLength climbs.

2. Take a snapshot during the stall

   Produce a dump and inspect thread stacks:

   dotnet-dump collect --process-id <pid>
   dotnet-dump analyze <dumpfile> -- "clrthreads"

   Search for patterns like System.Threading.Tasks.Task.InternalWait, System.Threading.WaitHandle.WaitOne, or System.Threading.Monitor.Enter.

3. Audit code for blocking constructs

   Search the solution for:

   • \.Result\b
   • \.GetAwaiter\(\)\.GetResult\(\)
   • \.Wait\(\)
   • Thread.Sleep\b
   • Console.ReadLine\b (in request path)

   Pay special attention to library calls that are synchronous by default (e.g., ADO.NET SqlCommand.ExecuteReader()).

4. Check lock hotspots

   Enable the lock contention counter:

   dotnet-counters monitor --process-id <pid> System.Runtime--ContentionRate

   A sustained high contention rate indicates excessive locking.

Fixes tied to findings

• Eliminate sync‑over‑async: Replace blocking calls with await and ensure the call chain is async all the way to the I/O layer (e.g., use await SqlCommand.ExecuteReaderAsync()).
• Offload CPU‑bound work: Wrap heavy computations in Task.Run(() => Compute(...)) or use a dedicated background service so they don’t consume request‑handling threads.
• Reduce lock scope: Keep critical sections as small as possible; consider ReaderWriterLockSlim or lock‑free structures like ConcurrentQueue<T> when appropriate.
• Temporary thread‑pool tuning: As a short‑term mitigation, raise the minimum worker threads:

ThreadPool.SetMinThreads(Environment.ProcessorCount * 2, 1000);

Risk: This masks the symptom, increases memory usage and context‑switch overhead, and does not address the root cause. Remove the call once the blocking code is fixed.
Escalation criteria
If latency remains high after:

All sync‑over‑async calls have been converted to async.
CPU‑bound work is moved off the request path.
Lock contention is minimized.

then consider:

Runtime configuration: Verify server GC (dotnet publish -c Release -r linux-x64 --self-contained false /p:ServerGarbageCollection=true) and that the process is not hitting container CPU limits that throttle thread injection.
Downstream timeouts: External services with long timeouts can keep threads blocked; add client‑side timeouts and consider circuit‑breaker patterns.
Queue‑based load leveling: Introduce a message queue (e.g., RabbitMQ, Azure Service Bus) to smooth bursts so the ThreadPool sees a steadier workload.

Verification

Reproduce the load with a tool such as wrk or bombardier while watching the counters from step 1. The desired outcome is a flat latency curve and a stable ThreadPool‑ThreadCount.
After applying fixes, capture another dump during the same load and confirm that no threads appear in WaitOne or InternalWait stacks.
Add a regression check in CI: fail the build if the average pending work‑item count exceeds a threshold (e.g., 100 items) during a short load test.

Limitations and practical checks
The thread‑pool injection rate (approximately one new thread per 500 ms) varies across .NET versions; consult the release notes for your target runtime. In containerized environments, the host’s CPU quota can make the pool appear starved even when the code is healthy—monitor the container’s CPU throttling metrics alongside the ThreadPool counters.