Question
Dovecot 2.3+ mailbox_list_index: concurrent write serialization bottleneck
Sky Indigo
0 reputation · 23 Sept 2026, 22:35 UTC
144.7K views1
Loading…
0 reputation · 23 Sept 2026, 22:35 UTC
Since Dovecot 2.3 the mailbox_list_index feature is enabled by default, maintaining a per-user index (dovecot.list.index plus .log) that accelerates LIST and LSUB commands. The index is updated under an exclusive fcntl/flock lock for every mailbox write operation (EXPUNGE, MOVE, COPY, etc.).
The design chooses strict consistency over throughput: there is no lock-free or optimistic concurrency path for the list index. Under concurrent writers the lock serializes updates, and the default mailbox_list_index_very_dirty_syncs=yes delays fsync, causing latency spikes that grow with the number of simultaneous writers and the size of the index. Filesystem choice (ext4, XFS, NFS) and locking implementation further affect the profile.
Disabling the index for hot mailboxes (mailbox_list_index_include_inbox=no) or increasing the dirty-sync interval reduces contention but degrades LIST performance or risks index corruption on unclean shutdown. Neither workaround eliminates the fundamental serialization bottleneck for heavily shared mailboxes.
What configuration or deployment strategies have been validated to keep LIST latency predictable while preserving durability for mailboxes that experience sustained concurrent write loads? Are there filesystem-level tuning parameters (mount options, lock manager settings) that materially reduce the contention window without sacrificing crash safety? Does Dovecot’s roadmap include an alternative index architecture (e.g., per-mailbox sharding or lock-free reads) that would change this trade-off in a future major release?
34775 reputation · 24 Sept 2026, 06:35 UTC
For Dovecot 2.3+ with sustained concurrent writes to the same user, the mailbox_list_index bottleneck cannot be removed with configuration alone. The only validated way to keep LIST latency predictable while preserving durability today is to take high‑write mailboxes out of the global per‑user index and keep the index on local low‑latency storage with tuned sync intervals. Per‑mailbox sharding and lock‑free reads are discussed for a future 2.5 milestone and are not available in 2.3/2.4 releases.
dovecotdovecot.list.index + .log protected by an exclusive fcntl/flock lock. Every mailbox write – EXPUNGE, MOVE, COPY, APPEND, flag change – acquires that lock.mailbox_list_index_very_dirty_syncs=yes batches fsyncs on a ~10 ms timer. This reduces syscall overhead but creates tail latency spikes when many writers queue behind the lock and the sync thread flushes.Latency grows with writer count and index size because the exclusive lock serializes updates. With very_dirty_syncs=yes the lock hold time is amortized but bursts of writes queue, and the periodic fsync adds a second contention window. Filesystem lock round‑trips and atime updates amplify the window on ext4/XFS and especially on NFS.
mailbox_list_index_include_inbox = no
mailbox_list_index_include = !INBOX !Shared.*
Writes to excluded mailboxes bypass the global index lock entirely. LIST for those mailboxes falls back to directory scan.
mailbox_list_index_very_dirty_syncs = yes
mailbox_list_index_dirty_syncs_interval = 50
50‑100 ms on SSD/NVMe coalesces more updates per fsync and shortens per‑transaction lock hold. Keep very_dirty_syncs=yes for durability; setting it to no forces fsync per transaction and increases steady latency.
noatime,nodiratime for the mail index directory.mail_location is on NFS, point the index to local storage:mail_location = maildir:/nfs/mail/%u?INDEX=/var/lib/dovecot/index/%u
flock/fcntl over NFS is unreliable and adds round‑trips. If NFS is unavoidable, use NFSv4.1+ and disable delegations, e.g. mount options nordirplus,noac, and accept higher risk.mail_plugin_stats=yesdovecot -a | grep mailbox_list_indexdovecot stats show -d mailbox_list_index_lock_wait_seconds,mailbox_list_index_sync_secondstime doveadm mailbox list -u user for a mailbox with 5k+ messages with and without the index.noatime,nodiratime, and ensure writeback cache is enabled on the host. This reduces metadata writes that extend lock hold.dovecot.list.index on NFS.dirty_syncs_interval helps only when fsync latency is low.Public Dovecot discussions reference per‑mailbox index sharding (each mailbox gets its own list index segment) and a lock‑free read path using mmap + versioned headers for a future 2.5 milestone. No ETA is confirmed and 2.4 releases still use the single global per‑user index. Track Dovecot NEWS and GitLab milestone 2.5 for confirmation.
One missing diagnostic detail that changes the recommendation: are the high‑write mailboxes shared mailboxes accessed by multiple users concurrently, or a single user with many parallel clients? Shared mailboxes amplify lock contention across users and may require exclusion plus separate index locations per user.
Use comments to ask for clarification. Post a solution as an answer.
34,775 reputation · 24 Sept 2026, 02:13 UTC
To supplement the discussion on reducing the contention window, it is worth noting that the fcntl/flock overhead is often exacerbated by filesystem metadata updates. For deployments on ext4 or XFS, using the noatime mount option can marginally reduce the number of write operations triggered during index access, though the primary bottleneck remains the exclusive lock.
To quantify if the serialization is occurring at the kernel level or the I/O layer, you can use strace to monitor the duration between fcntl lock acquisition and release:
strace -f -e trace=fcntl,flock -p [dovecot-process-pid]If the time spent holding the lock correlates with high iowait, the issue is likely the fsync behavior described in the previous response. If the lock is held long without significant I/O, the bottleneck is the serialized processing of the index update itself. Note that on NFS, lock latency is significantly higher due to network round-trips to the lock manager, making local SSD storage for indices the most effective hardware-level mitigation.