Preventing Resource Exhaustion with Counting Semaphores

The most common bottleneck in high-concurrency applications isn't CPU speed—it's resource exhaustion. When an application spins up hundreds of threads to query a database or API that only supports twenty concurrent connections, the result is usually a cascade of timeouts, connection pool exhaustion, or total system failure. While developers often use mutexes to protect shared state, they frequently need a mechanism to limit the volume of active workers hitting a finite resource.

A Semaphore (often referred to as a Sema) is a concurrency primitive designed for this specific scenario. Unlike a mutex, which allows exactly one thread at a time, a semaphore maintains a set of permits. It acts as a gatekeeper, ensuring that your system never exceeds its defined capacity.

The Permit Mechanism

A semaphore operates on an internal counter. When a thread wants to access a protected resource, it must call acquire(). If a permit is available, the counter decrements and the thread proceeds. If the counter is zero, the thread blocks (pauses) until another thread calls release(), which increments the counter and wakes up a waiting thread.

There are two primary configurations used in engineering:

• Counting Semaphores: Initialized with a value greater than one. These are ideal for managing resource pools, such as limiting the number of simultaneous file handles or database connections.
• Binary Semaphores: Initialized with exactly one permit. While they behave similarly to mutexes, they lack "ownership"; any thread can release a binary semaphore, regardless of which thread acquired it.

Worked Example: Throttling API Requests

Consider a service that fetches data from a third-party API that limits you to five concurrent requests. To avoid being rate-limited or banned, you can use a counting semaphore to enforce this limit locally.

import java.util.concurrent.Semaphore;

public class ApiThrottler {
    // Limit to 5 concurrent requests
    private final Semaphore semaphore = new Semaphore(5);

    public void fetchData(String endpointId) {
        try {
            // Request a permit; blocks if 5 are already active
            semaphore.acquire();
            
            // Perform the network operation
            System.out.println("Fetching: " + endpointId);
            performNetworkCall(endpointId);
            
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        } finally {
            // ALWAYS release in a finally block to prevent permit leaks
            semaphore.release();
        }
    }

    private void performNetworkCall(String id) {
        try { Thread.sleep(1000); } catch (InterruptedException e) {}
    }
}

In this implementation, the finally block is critical. If the network call throws an exception and release() is not called, that permit is lost. Over time, the permit count will hit zero, and the application will hang indefinitely—a condition known as a permit leak.

Trade-offs and Engineering Risks

While semaphores are powerful for throttling, they introduce specific risks that require careful management:

• Deadlocks: If a thread requires permits from two different semaphores to proceed, and two threads acquire them in opposite orders, the system will freeze. Always acquire multiple semaphores in a consistent, predefined order.
• Contention Overhead: If hundreds of threads fight for a small number of permits, the operating system spends significant CPU time on context switching (swapping threads in and out of the CPU). In extreme high-load scenarios, a queue-based worker pattern is more efficient than blocking threads.
• Priority Inversion: Because semaphores do not track which thread holds a permit, a high-priority thread can be blocked behind a low-priority thread that entered the critical section first.

Verifying the Throttle

To verify that your semaphore is correctly limiting concurrency, launch a number of threads that exceeds your permit limit (e.g., 20 threads for a 5-permit semaphore). Use a concurrent counter or log timestamps to track active entries. If the implementation is correct, the number of threads executing the performNetworkCall method simultaneously should never exceed five.