Diagnosing Java Semaphore stalls, leaks, and starvation

Threads stop making progress and thread dumps show many threads parked in java.util.concurrent.Semaphore.acquire(). The application looks deadlocked, but no explicit lock is held. This is usually semaphore exhaustion: permits are not being returned at the rate they are taken.

A Semaphore is a counting permit gate. acquire() blocks until a permit is available; release() returns one. When the count reaches zero and never recovers, the service stalls.

Recognizable conditions

Stall under load: latency climbs, request threads pile up, CPU is low. Dumps show WAITING on Semaphore.acquire.

Slow degradation: permits trend down over hours. availablePermits() logs show a monotonic decline to zero.

Intermittent race errors: code proceeds without a permit because tryAcquire returned false but was treated as success.

Starvation of specific workers: some threads never acquire despite idle permits, common with long-running holders under unfair mode.

Cause and diagnostic table

Symptom	Likely cause	Quick check
Threads block indefinitely on acquire()	Permits exhausted	availablePermits() == 0 and queue length growing in dump
Permits slowly decrease to zero over time	Permit leak: release missed on exception path	availablePermits() below initial permits after steady state
Some threads never get permits while others repeatedly succeed	Unfair acquisition, starvation	Semaphore constructed with fair=false, holders hold permits long
Race condition / data corruption under load	tryAcquire misused, release called without successful acquire	Code releases unconditionally after tryAcquire

Ordered checks

1. Confirm blocking with a thread dump

Run jstack <pid> > /tmp/dump.txt from a host with OS permission to attach to the JVM process.

Look for stacks containing java.util.concurrent.Semaphore.acquire and AbstractQueuedSynchronizer.acquireQueued. Count unique threads in that state. Risk: jstack pauses the target briefly on some JVMs.

2. Inspect permit count at runtime

Expose availablePermits() via logging or JMX. Do not rely on a single sample; log periodically.

log.info("semaphore permits={}, queue={}", sem.availablePermits(), sem.getQueueLength());

A snapshot is not a guarantee of future state, but a persistent low value correlates with exhaustion.

3. Review acquire/release pairing

Search code for every acquire and tryAcquire call on the same Semaphore instance. Verify each successful acquire has a matching release in a finally block.

Anti-pattern:

sem.acquire();
doWork(); // throws
sem.release();

Correct pattern:

sem.acquire();
try {
  doWork();
} finally {
  sem.release();
}

4. Audit tryAcquire usage

Check that the return value is tested before proceeding and that release is only called when acquire succeeded.

if (sem.tryAcquire(1, TimeUnit.SECONDS)) {
  try { doWork(); } finally { sem.release(); }
} else {
  // handle failure, do not proceed
}

5. Determine fairness configuration

Check constructor: new Semaphore(n) is unfair. new Semaphore(n, true) is fair.

Fair mode preserves FIFO order but adds throughput cost. Use it only when ordering matters.

Fixes tied to findings

If permits exhausted and dumps show blocking: ensure release is in finally. Add defensive logging of permit count after release to detect negative drift.

If permit leak confirmed: fix missing finally paths, and for existing leaks, consider a controlled restart after code fix. Do not call release extra times to compensate; that can inflate permits beyond the intended limit.

If starvation observed under unfair mode: evaluate whether ordering is required. If yes, recreate Semaphore with fair=true after a maintenance window, as fairness changes affect queue semantics. If no, reduce hold time or increase permits.

If tryAcquire misuse found: gate work behind the boolean result and avoid unconditional release. Add a unit test that acquires all permits, attempts another acquire with timeout, and asserts false return and no permit leak.

Escalation criteria

Escalate when:

• Permit count is zero and thread dumps show >50% of request threads blocked for >5 minutes with no recovery.
• Code review shows shared Semaphore instance used across unrelated subsystems, risking cross-domain exhaustion.
• Fairness change or permit limit increase is required in production and impacts throughput SLAs.

Limitations and practical check

availablePermits() is a point-in-time view and can change immediately after read. Thread dumps do not show historical permit flow. Semaphore does not provide built-in leak detection. Verification requires code inspection plus runtime observation.

Check the result practically by logging permit count before and after a representative load burst and confirming it returns to baseline.