Unable to lock store directory: [path] – Neo4j storage subsystem lock acquisition behavior

Question

Goal: Clarify whether Neo4j’s storage layer should automatically retry obtaining an exclusive lock on its data directory when a transient NFS latency causes the initial lock attempt to fail, or whether it should treat the failure as fatal and shut down immediately.

Constraints: The behavior varies between Neo4j 3.x, 4.x, and 5.x releases and can be influenced by the dbms.lock.acquisition.timeout setting; logs show the same “Unable to lock store directory” message regardless of outcome, making it difficult to discern if a retry is pending without additional instrumentation.

Questions: Does Neo4j currently implement a retry loop for lock acquisition, and if so, what is the default retry interval and maximum attempts? How does the dbms.lock.acquisition.timeout parameter interact with any internal retry logic across different versions?

Siddharth · Answer

Neo4j does not implement an infinite retry loop for the initial store directory lock. If the storage subsystem cannot acquire an exclusive lock within the period defined by the timeout settings, it treats the failure as fatal and shuts down.
Lock Acquisition Mechanism
p>The "Unable to lock store directory" error occurs during the database bootstrap phase. Neo4j attempts to acquire an exclusive lock on the data directory (typically via a store.lock file) to prevent data corruption from concurrent instances. The behavior regarding transient latency depends on the version and specific configuration:

Neo4j 3.x and 4.x: These versions rely heavily on a fixed timeout. If the lock is not granted due to NFS latency or process contention within that window, the process logs the error and terminates.
  Neo4j 5.x: While the core logic remains similar, the storage engine is more sensitive to file system behavior. However, it still will not retry indefinitely if the initial lock timeout is exceeded.

Interaction with dbms.lock.acquisition.timeout
The dbms.lock.acquisition.timeout parameter (or its equivalent in older neo4j.properties files) defines the maximum duration the engine will wait for the lock to become available.

The Timeout vs. Retry Interval: This parameter is not a retry interval (e.g., "retry 5 times every 2 seconds"). It is a total duration window for the acquisition attempt.
  NFS Latency: On NFS, high latency can cause the kernel-level lock request to take longer than the dbms.lock.acquisition.timeout value, causing Neo4j to fail.

Resolution and Diagnostic Steps
If you are experiencing transient failures on a networked filesystem, follow these steps to isolate the cause:

Identify Existing Locks: Check if another process is actually holding the directory.
    lsof | grep /path/to/neo4j/data
  
  Check for Stale Lock Files: If Neo4j crashed, a store.lock file might remain. Only delete this if you are certain no Neo4j process is running.
  Increase the Timeout: If the NFS environment is inherently slow, increase the timeout to allow more time for the file system to respond.
    # dbms.lock.acquisition.timeout=60s

Diagnostic Question: To provide more specific advice, are you running Neo4j on a local SSD or a networked NFS/SMB mount?

Unable to lock store directory: [path] – Neo4j storage subsystem lock acquisition behavior

1 answer

Neo4j does not implement an infinite retry loop for the initial store directory lock. If the storage subsystem cannot acquire an exclusive lock within the period defined by the timeout settings, it treats the failure as fatal and shuts down.

Lock Acquisition Mechanism

Interaction with `dbms.lock.acquisition.timeout`

Resolution and Diagnostic Steps

Your perspective belongs here.

1 question comment