Diagnosing and Resolving OutOfMemoryException in .NET Managed Heaps

The Allocation Failure Problem

An OutOfMemoryException (OOM) in .NET does not always mean the system has run out of physical RAM. More often, it indicates that the Common Language Runtime (CLR) cannot find a contiguous block of memory large enough to satisfy a specific allocation request. This typically happens due to managed heap exhaustion or fragmentation within the Large Object Heap (LOH).

Identifying the OOM Variant

Before applying a fix, you must determine if the failure is caused by a genuine leak (growing heap) or fragmentation (gaps in the heap).

Symptom	Likely Cause	Diagnostic Indicator
Steady increase in memory over time	Managed Memory Leak	Gen 2 size grows continuously; GC cannot reclaim space.
Sudden OOM despite available RAM	LOH Fragmentation	High LOH size but low object count; frequent large array allocations.
OOM immediately upon startup/load	Address Space Exhaustion	Application running as x86 (32-bit) process on a 64-bit OS.

Step-by-Step Diagnostic Workflow

Perform these checks in order to isolate the memory pressure source. These steps assume .NET 6+ and the installation of .NET CLI tools.

1. Verify Process Architecture

   Ensure the application is targeting x64. A 32-bit process is limited to 2GB–4GB of virtual address space regardless of physical RAM.

   Run this on the host machine to check the process architecture:

   # Windows PowerShell
   Get-Process -Name YourAppName | Select-Object ProcessName, @{Name="Architecture"; Expression={if($_.MainModule.ModuleName -like "*x86*") {"x86"} else {"x64"}}}

2. Monitor Real-Time GC Metrics

   Use dotnet-counters to observe if the heap is growing linearly or if Gen 2 collections are failing to reduce memory usage.

   # Run from terminal with administrative/sudo permissions
   dotnet-counters monitor -p [PID] --counters System.Runtime

   What to look for: If "GC Heap Size" increases while "Gen 0/1/2 Size" remains high after a full GC, you have rooted objects preventing collection.

3. Analyze the Heap Snapshot

   If metrics confirm a leak, capture a dump to find the "root" (the object holding the reference that prevents GC).

   # Capture dump (requires dotnet-dump tool)
   dotnet-dump collect -p [PID]

   Open the dump and run dumpheap -stat to see which object types occupy the most memory.

Fixes Based on Findings

Scenario A: Managed Memory Leak

If dumpheap shows millions of instances of a specific class, check for common rooting patterns:

• Static Collections: Static lists or dictionaries that grow indefinitely.
• Event Handlers: Long-lived objects subscribing to events on short-lived objects without unsubscribing.
• Captured Variables: Closures in lambda expressions holding onto large objects.

Scenario B: LOH Fragmentation

Objects larger than 85,000 bytes are placed in the Large Object Heap (LOH). Unlike the small object heap, the LOH is not compacted by default, leaving "holes" that cannot be used for new large allocations.

The Fix: Implement LOH compaction for critical cleanup phases. This is an expensive operation and should not be done frequently.

// Run this before a known high-pressure allocation phase
GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
GC.Collect();

Scenario C: High Allocation Rate (Churn)

If the OOM is triggered by high-frequency allocation of large buffers (e.g., byte arrays for file processing), avoid the heap entirely using ArrayPool.

// Instead of: byte[] buffer = new byte[1024 * 100];
// Use the Shared Pool:
byte[] buffer = ArrayPool<byte>.Shared.Rent(1024 * 100);
try {
    // Use buffer
} finally {
    ArrayPool<byte>.Shared.Return(buffer);
}

Runtime Configuration Tuning

For high-throughput server applications, the gcServer setting in runtimeconfig.json determines if the GC uses a separate heap per CPU core. While this improves performance, it increases the total memory footprint.

{
  "runtimeOptions": {
    "configProperties": {
      "System.GC.Server": true
    }
  }
}
Risk: If your application is memory-constrained (e.g., a small Docker container), setting gcServer: false (Workstation GC) can reduce the memory overhead at the cost of some throughput.

Verification and Limitations
To verify the fix, run the application under a load test while monitoring dotnet-counters. The "GC Heap Size" should plateau rather than climb linearly.
Limitations: LOH compaction and GC.Collect() cause "Stop-the-World" pauses, which will spike latency. Always prefer ArrayPool or Memory<T> over manual GC triggers.

Rollback Plan
If changes to runtimeconfig.json cause instability or unexpected latency spikes, revert the System.GC.Server value to its previous state and restart the process.