Using Semaphores to Throttle Concurrent API Calls

When a service spikes to hundreds of requests per second, the limiting factor is often not CPU but the downstream system it calls—such as a REST API that allows only a few concurrent connections. Blindly flooding that endpoint leads to HTTP 429 errors, latency spikes, or even temporary bans.

Why a Semaphore Fits the Problem

A Semaphore maintains a counter of permits. Threads call acquire() to take a permit; if none are available they block until another thread calls release(). Unlike a mutex, which ties ownership to the locking thread, any thread may release a permit, making the primitive ideal for throttling access to a shared pool.

Worked Example: Limiting Three Concurrent Calls

The following Java‑like sketch shows a throttler that caps outbound calls to three at a time. Run this code in a Java ≥ 8 environment with the java.util.concurrent class on the classpath. No special permissions are required beyond normal JVM execution.

import java.util.concurrent.Semaphore;

public class ApiThrottler {
    private final Semaphore permits;

    public ApiThrottler(int maxConcurrent) {
        this.permits = new Semaphore(maxConcurrent);
    }

    public void callApi(String id) {
        try {
            permits.acquire(); // blocks if maxConcurrent calls are active
            System.out.println(\"[\" + Thread.currentThread().getId() + \"] processing \" + id\");
            // Simulate network latency; replace with actual HTTP client call
            Thread.sleep(1500);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt(); // preserve interrupt status
        } finally {
            permits.release(); // always return the permit
        }
    }
}

If ten threads invoke callApi simultaneously, the first three proceed immediately; the remaining seven block. As each thread finishes its simulated work and hits the finally block, a permit is returned and the next waiting thread is unblocked.

Trade‑offs and Limitations

The primary risk is a “leaked permit”: if an exception occurs before the release() in the finally block, the permit is never returned, eventually exhausting the pool and causing a deadlock. Ensure every acquire() is paired with a release() in a finally block or use a try‑with‑resources wrapper.

Another limitation is that a Semaphore does not track which thread holds a permit. Accidentally calling release() more times than acquire() inflates the counter beyond the initial capacity, weakening the throttling guarantee. Defensive code can check the current permit count via permits.availablePermits() during development or testing.

Verification Checklist

1. Run the throttler with a fixed number of threads (e.g., 10) and a Semaphore of size 3.
2. Observe that no more than three callApi executions overlap in time (you can log timestamps).
3. Confirm that after all threads finish, permits.availablePermits() returns the original value (3).
4. Introduce a deliberate missing release() in a test thread and verify that the available permits drop and subsequent calls block indefinitely.

These steps give confidence that the semaphore is behaving as a resource throttle rather than a mutex.

When you need to limit concurrent access to an external service, reach for a Semaphore with a size matching the service’s capacity. Pair each acquire() with a guaranteed release(), monitor the available permits during testing, and you’ll avoid both overload and unnecessary queuing.