Lazy index loading in Dovecot for large mailboxes

Problem: Slow IMAP startup with large mailboxes

When a mailbox contains tens of thousands of messages, Dovecot traditionally reads the entire index file (.idx) into memory at process start. This can cause noticeable latency, especially on servers with limited RAM or slow storage, and may delay the first IMAP command after a restart.

How Dovecot loads indexes lazily

Starting with version 2.3, Dovecot enables lazy index loading by default. Instead of reading the whole .idx file, it memory‑maps the file with mmap. The kernel pages in only the parts that are actually accessed, so folder‑level metadata is available immediately while individual message records are fetched on demand. For mbox mailboxes the same principle applies, but because mbox stores only a single file per folder, lazy loading works at the folder level only.

Configuring lazy loading and cache size

Two settings control the behaviour:

• mail_cache_size – maximum amount of memory Dovecot may use for caching index pages. A larger value reduces disk I/O for frequently accessed messages.
• mail_temp_dir – directory used for temporary files during index rebuilds; placing it on a fast filesystem can shorten the rebuild window.

Example snippet for /etc/dovecot/conf.d/10-mail.conf:

mail_cache_size = 256M
mail_temp_dir = /var/tmp/dovecot

After changing the file, reload Dovecot:

sudo systemctl reload dovecot

Worked example: Verify lazy loading and rebuild a stale index

1. Check whether lazy loading is active for a user’s mailbox:

   sudo doveadm debug mailbox -u [contact removed] INBOX

   Look for output lines that mention mmap or lazy. If the command returns “index file is mapped”, lazy loading is in effect.

2. If the index appears stale (timestamps older than the mailbox file) or you notice slowdowns, rebuild it:

   sudo doveadm index -u [contact removed] INBOX

   This operation reads the mailbox, creates a new .idx file, and replaces the old one. During the rebuild the mailbox is briefly locked, so concurrent IMAP clients may see a temporary pause.

3. Verify the rebuild succeeded:

   sudo doveadm debug mailbox -u [contact removed] INBOX

   The timestamp of the .idx file should now be close to the current time, and the lazy‑loading message should still appear.

Trade‑off and limitation

The main benefit of lazy loading is reduced startup latency and lower initial memory footprint. The trade‑off is increased page‑fault activity when clients access scattered messages, which can raise I/O if the underlying storage is slow. Additionally, mmap‑based lazy loading fails on filesystems that do not support shared memory mapping (e.g., some network filesystems). In such cases Dovecot falls back to reading the whole index, negating the benefit and increasing startup time.

You can test whether mmap is working by checking the kernel’s /proc/<pid>/maps for the Dovecot process; a line showing the .idx file with rw‑p indicates a successful mapping.

Actionable closing

If you manage large mailboxes, enable lazy loading (default in recent Dovecot), tune mail_cache_size to match your workload, and place mail_temp_dir on fast storage. Periodically run doveadm debug mailbox to confirm that indexes are mapped and not falling back to full reads. When you notice stale indexes, schedule a doveadm index run during a low‑traffic window and keep a backup of the .idx files so you can roll back if the rebuild introduces unexpected issues.