Distributed Semaphore for Microservice Rate‑Limiting: Architecture, Design, and Operational Guidance

Problem & Takeaway

When many stateless microservices need to enforce a global concurrency limit—such as limiting the number of simultaneous external API calls—each service must coordinate with the others to avoid over‑granting permits. The key requirement is atomic acquisition and release of a shared counter, without a separate lock that could become a bottleneck. A Redis‑backed Lua script provides the smallest, most reliable design that satisfies these constraints.

Requirements

• Atomic acquire and release operations across all services.
• Maximum concurrent permits N configurable at deployment time.
• Visibility of the current counter value for monitoring.
• Graceful degradation on Redis unavailability.
• Isolation of semaphore state from application data.

Minimal Suitable Design

The design uses a single Redis key to store the counter and a Lua script that performs the increment/decrement atomically. Redis’ EVAL command guarantees that the script runs without interruption, so the counter can never become inconsistent.

-- Lua script: sema_acquire.lua
-- KEYS[1] – semaphore key
-- ARGV[1] – maximum permits N
-- ARGV[2] – lease timeout in seconds
local current = tonumber(redis.call("GET", KEYS[1]) or "0")
local max = tonumber(ARGV[1])
if current < max then
  redis.call("INCR", KEYS[1])
  redis.call("EXPIRE", KEYS[1], ARGV[2])
  return 1
else
  return 0
end

For release, a second script decrements the counter only if it is greater than zero.

-- Lua script: sema_release.lua
-- KEYS[1] – semaphore key
local current = tonumber(redis.call("GET", KEYS[1]) or "0")
if current > 0 then
  redis.call("DECR", KEYS[1])
  return 1
else
  return 0
end

Each microservice calls EVALSHA with the script SHA, the semaphore key (e.g., sema:api_calls), and the arguments. The lease timeout protects against orphaned permits if a service crashes after acquiring.

Trust / Data Boundaries

• The semaphore key is the only data exposed to services; all other application data resides in separate databases.
• Clients never see the raw counter value; they only receive a success/failure response.
• Redis replication and ACLs can be used to restrict which services can modify the semaphore key.

Operational Checks

1. Counter Drift Monitoring
   Periodically query GET sema:api_calls and compare against a local snapshot or expected value. Alert if the counter deviates by more than 5% of N.
2. Acquisition Latency
   Instrument the acquire call to record latency. A sudden spike may indicate network partition or Redis slowdown.
3. Zero / Over‑Max Alerts
   If the counter reaches 0 (no permits available) or exceeds N (bug or partition), trigger an alert for manual inspection.

Failure Modes & Recovery

Mode	Impact	Recovery
Redis Outage	All acquire attempts fail; services may block or throttle.	Fallback to in‑memory limiter with exponential back‑off; automatically retry once Redis recovers.
Network Partition	Stale counters; orphaned permits may accumulate.	Lease timeouts reclaim permits; monitor for counter > N and manually reset if needed.
Script Bug	Counter corruption; over‑granting or deadlock.	Unit‑test scripts; CI pipeline runs redis-cli EVAL against a test instance before deployment.

When to Change the Design

• High Availability Requirement – If a single Redis instance is unacceptable, deploy a Redis Cluster or Sentinel with failover and re‑balance the counter key across shards.
• Security‑Critical Controls – For controls that must never be bypassed, add a separate audit log in a tamper‑evident store and enforce strict ACLs on the semaphore key.
• Multi‑Region Deployment – If services span regions, consider using a globally replicated datastore or a consensus protocol (e.g., etcd) to avoid cross‑region latency spikes.
• Dynamic Thresholds – When N changes at runtime, expose an API to update the script arguments; ensure all services reload the new value atomically.

Example Deployment Snippet

Assuming Docker Compose, the Redis service and a simple Go microservice that uses the semaphore are defined as follows:

# docker-compose.yml
version: "3.9"
services:
  redis:
    image: redis:7
    ports:
      - "6379:6379"
    command: ["redis-server", "--save", "", "--appendonly", "no"]
  api-service:
    build: ./api-service
    environment:
      - REDIS_URL=redis://redis:6379
      - SEMAPHORE_KEY=sema:api_calls
      - MAX_PERMITS=5
      - LEASE_SECONDS=30
    depends_on:
      - redis

In Go, the acquire call would look like:

func AcquirePermit(client *redis.Client, key string, max int, lease int) (bool, error) {
    script := redis.NewScript(`
        local current = tonumber(redis.call("GET", KEYS[1]) or "0")
        local max = tonumber(ARGV[1])
        if current < max then
            redis.call("INCR", KEYS[1])
            redis.call("EXPIRE", KEYS[1], ARGV[2])
            return 1
        else
            return 0
        end
    `)
    res, err := script.Run(context.Background(), client, []string{key}, max, lease).Result()
    if err != nil { return false, err }
    return res.(int64) == 1, nil
}

Verification Checklist

• Spin up 10 concurrent clients and confirm the counter never exceeds N.
• Stop Redis; verify acquire attempts return a timeout and services fallback correctly.
• Simulate a network partition; ensure lease timeouts reclaim permits and the counter returns to N after healing.

Limitations

• Redis single‑point failure unless replicated; add Sentinel or Cluster for HA.
• Lease timeout may be insufficient if a service crashes after acquiring; consider a background cleanup job.
• Lua scripts must be versioned; if the script changes, all services need to reload the new SHA.

Conclusion

A Redis‑backed Lua semaphore delivers atomic, low‑latency rate limiting with minimal infrastructure. By clearly defining trust boundaries, operational checks, and recovery paths, teams can confidently deploy this pattern in production microservice architectures.