Use Consul KV Sessions as Distributed Locks for Singleton Jobs

The Problem: Duplicate Cron Jobs Across Worker Instances

You have a nightly ingestion job running on three identical workers. The job must run exactly once, but your scheduler triggers it on every instance. Teams often solve this by adding Redis, a database lock row, or a separate leader-election service. If you already operate Consul for service discovery, you can use its KV store and sessions as a distributed lock instead of provisioning new infrastructure.

The Useful Takeaway

A Consul session is a lease with a time-to-live (TTL). Pair it with a KV key and the ?acquire= query parameter so only one holder can own the key at a time. Renew the session while the job runs, then release it. Consul serializes the write through Raft, so the acquire result is authoritative even if ordinary reads are stale.

Core Mechanics: Session, Key, Renew, Release

Create a session

Call PUT /v1/session/create with a JSON body that sets TTL and LockDelay explicitly. The session behaves like a lease: if it is not renewed before the TTL expires, Consul invalidates it and any lock it holds becomes available after LockDelay.

Acquire the lock

Call PUT /v1/kv/<lock-key>?acquire=<session-id>. Consul returns true only when the key is free or already held by that session. Every other caller receives false. Do not decide ownership from a plain GET; reads can be stale unless you request consistency.

Renew and release

Renew the session on a timer, for example every one-third of the TTL, using PUT /v1/session/renew/<session-id>. When the job finishes, call PUT /v1/kv/<lock-key>?release=<session-id> or destroy the session. Destroying the session is the closest thing to a rollback: it immediately gives up the lock.

Worked Example: Two Clients Competing for One Lock

Run this on a laptop or throwaway VM where you can bind port 8500. No root is required, but the user must be able to start a network listener.

# Start a single Consul dev agent
consul agent -dev -client 127.0.0.1

In a second terminal, Client A creates a session with a 15-second TTL and release behavior:

curl -X PUT -d '{"Name":"job-lock","TTL":"15s","Behavior":"release"}' http://127.0.0.1:8500/v1/session/create

The response contains an ID such as a1b2c3d4. Client B repeats the call and gets a different ID, such as e5f6g7h8. Both clients now try to acquire the same key:

# Client A
curl -X PUT "http://127.0.0.1:8500/v1/kv/job/singleton?acquire=a1b2c3d4"

# Client B
curl -X PUT "http://127.0.0.1:8500/v1/kv/job/singleton?acquire=e5f6g7h8"

Only the first request returns true. The holder renews the session every five seconds:

curl -X PUT http://127.0.0.1:8500/v1/session/renew/a1b2c3d4

After the job completes, the holder releases:

curl -X PUT "http://127.0.0.1:8500/v1/kv/job/singleton?release=a1b2c3d4"

Client B can retry the acquire. If Client A's agent stops, the session expires after its TTL and the next acquire succeeds once LockDelay elapses.

Fencing: Treat the Lock as Advisory, Not Authoritative

A successful acquire proves you won the Consul lock at that moment. It does not prove the holder is still alive. After acquiring, read the key's ModifyIndex and pass it to the guarded resource; have that resource reject older indices. You can also protect a separate epoch record with ?cas= so updates are rejected if the underlying key changed.

Trade-offs and Operational Limits

Every acquire, renew, and release is a Raft consensus write. That means lock churn adds latency and fsync load to the Consul server path. Use this pattern for long-running singleton jobs, not for high-frequency locking or as a general datastore. Keep KV values small; Consul KV is coordination storage, not a blob store.

Set TTL and LockDelay explicitly and measure failover in your environment. A TTL that is too short causes flapping; a LockDelay that is too long slows recovery after a crash. During a network partition, the holder may be uncertain whether it still owns the lock, so design jobs to be idempotent.

Verification Checklist

1. Run consul version and confirm the release you are testing; session fields and defaults vary across versions.
2. Start consul agent -dev and create a session, then acquire the same key from two clients. Confirm only one returns true.
3. Inspect response headers with curl -si on GET /v1/kv/job/singleton. Check X-Consul-Index, X-Consul-Knownleader, and X-Consul-Lastcontact, then repeat with ?consistent to compare stale versus consistent reads.
4. Force failover by destroying the holder's session: curl -X PUT http://127.0.0.1:8500/v1/session/destroy/a1b2c3d4. Watch consul session info <id> and measure how long LockDelay delays the next successful acquire.
5. On a non-dev agent with ACLs enabled, confirm acquire fails without a token and succeeds with a least-privilege token granting only session:write and kv:write on the lock prefix.

ACL Example

Create a policy scoped to the lock prefix:

acl policy create -name=job-lock -policy='
  path "job/singleton" {
    policy = "write"
  }
  session "job-lock" {
    policy = "write"
  }
'

Attach that policy to a token and use it with curl -H "X-Consul-Token: <token>". Without a token on an ACL-enabled agent, the acquire call should return 403.

Practical Closing

Consul KV sessions are a reasonable choice when you already run Consul and need a simple, advisory mutex for singleton jobs. Measure failover time, keep values small, and treat the lock as coordination rather than a source of truth for the guarded resource.