Solving IMAP Lag: Optimizing Dovecot Indexing for Large Mailboxes

The Problem: The "Syncing Folders" Hang

When an IMAP client connects to a mailbox containing tens of thousands of messages, the initial synchronization often hangs or feels sluggish. This happens because the server must determine the status of every message — checking flags, calculating sizes, and verifying UIDs — before the client can display the list.

Without an efficient indexing strategy, Dovecot is forced to perform a linear scan, opening thousands of individual files on disk for every session.

The takeaway: move the metadata burden from raw mail storage to Dovecot's index files, ensuring the server reads a single optimized index rather than thousands of fragmented mail files.

How Dovecot Indexing Works

Dovecot uses a separate index file system to store metadata. Instead of parsing a raw email to see if it is marked as \Seen, Dovecot looks up the message UID in a dovecot.index file. This transforms an O(n) operation into a near-constant time lookup.

For Maildir, where every email is a separate file, indexing is critical. The large number of inodes makes disk I/O a bottleneck on folder scans. mdbox packs multiple messages into a single file and is naturally more efficient, but still relies on indexes to avoid scanning the entire archive for a specific search query.

Optimizing for Search and Sort

IMAP SEARCH and SORT commands are computationally expensive without indexes. Dovecot's indexing allows the server to query metadata without opening every individual file.

The mail_preset feature lets the server pre-calculate message flags and sizes, reducing disk I/O during client synchronization and lowering time to first byte for folder summaries.

Implementation: Managing and Verifying Indexes

Index management is handled via doveadm. If a discrepancy appears between client view and disk, or after an unclean shutdown, metadata can be rebuilt.

Example: Forcing a Metadata Rebuild

Run on the mail server as root or a Dovecot administrative user:

doveadm force-resync -u user@example.com INBOX

Expected behavior: Dovecot scans the raw mail files in INBOX and rewrites dovecot.index and dovecot.index.cache. The next IMAP login should reflect the actual state of mail storage.

Risk: Running force-resync on a massive mailbox during peak hours can cause a temporary spike in disk I/O, potentially slowing other users on the same disk.

The Memory vs. Disk Trade-off

Dovecot's cache system stores frequently accessed message fragments in memory or fast disk storage to accelerate delivery of message bodies. Aggressive caching increases memory consumption per concurrent IMAP session.

Strategy	Benefit	Trade-off
Aggressive Indexing	Instant folder loads	Increased disk space for .index files
High Memory Caching	Fast message body retrieval	Higher RAM usage per user session
Lazy Indexing	Lower initial disk overhead	Slow initial IMAP syncs

Verification and Limitations

Verify indexing by checking the user's mail directory for dovecot.index and dovecot.index.cache files after login. Incorrect permissions, such as files owned by root instead of the vmail user, cause Permission Denied errors despite correct mailbox permissions.

Limitation: Indexing does not replace the need for fast underlying storage. Initial index creation remains slow on HDDs. A common pattern is to place index files on SSD or NVMe while keeping raw mail on slower storage.

Actionable Summary

If users report hanging folders: first check for the existence of dovecot.index files, ensure the vmail user can write to them, and use doveadm force-resync to clear corruption from unclean shutdowns.