Inside Dovecot's Mailbox Index: How dovecot.index, Cache, and Log Power Fast IMAP

The problem: IMAP without full message scans

When an IMAP client asks for FETCH ENVELOPE or FETCH BODYSTRUCTURE, a naïve server would open each raw message, parse MIME, and return the requested parts. On a mailbox with hundreds of thousands of messages that quickly becomes an I/O and CPU bottleneck. Dovecot solves this by keeping three small, per‑mailbox index files that together answer most IMAP commands without touching the original message files.

The three core files

• dovecot.index – metadata and a UID‑to‑offset map. It stores per‑message flags, size, and the physical location of the message in the maildir/mbox file.
• dovecot.index.cache – decoded header and body fields that the administrator has chosen to cache (e.g., ENVELOPE, BODYSTRUCTURE, RFC822.SIZE, custom headers). This file is read‑only for the IMAP process and is mmap()ed for zero‑copy access.
• dovecot.index.log – a write‑ahead log. Every flag change, expunge, or append is appended sequentially. Periodic checkpoints compact the main index, giving crash‑safe durability and providing the replication stream used by dsync.

All three files live inside the mailbox directory (or a separate index directory when INDEX= is set). Because each mailbox owns its indexes, there is no central database bottleneck, but it also means full‑text search requires a separate plugin (fts‑solr or fts‑lucene) that maintains its own indexes.

Worked example: tuning the cache for a typical workload

Assume a server where most clients fetch envelope and bodystructure, but rarely request full message bodies. The default mail_cache_fields includes a handful of headers; we can tighten it to avoid cache bloat:

# /etc/dovecot/conf.d/10-mail.conf
mail_cache_fields = ENVELOPE BODYSTRUCTURE RFC822.SIZE
mail_always_cache_fields = ENVELOPE RFC822.SIZE

With this configuration:

• ENVELOPE and RFC822.SIZE are always written to the cache at delivery time (low latency for the most common FETCH).
• BODYSTRUCTURE is cached on first demand, then reused.
• Large fields like BODY[] or BODY[TEXT] are never stored, keeping the cache file roughly 100–300 bytes per message.

To verify the cache contents for a specific mailbox, run:

doveadm dump index /var/vmail/example.com/user/Maildir/dovecot.index

The output lists the record count, UID range, and the exact cache fields stored for that mailbox. If you see unexpected fields, adjust mail_cache_fields and run doveadm force-resync -u user INBOX to rebuild.

NFS, replication, and upgrade gotchas

• NFS: mmap() and POSIX locks behave unpredictably on NFS. Place indexes on local storage with mail_location = maildir:/var/vmail/%d/%n:INDEX=/var/indexes/%d/%n. The /var/indexes tree should be owned by the vmail user, not root.
• dsync initial sync: The first replica build reads every raw message to create the target indexes. Plan bandwidth and time for large mailboxes; a 10 GB mailbox can take tens of minutes on a 1 Gbps link.
• Major version upgrades (2.2 → 2.3 → 3.0) may migrate the index format (v4/v5) on first access, causing a one‑time latency spike. Test the upgrade on a replica or a staging mailbox before rolling out to production.

Limitation: no global search index

The per‑mailbox indexes give fast per‑folder operations, but they do not provide a server‑wide search capability. If you need full‑text search across all folders, you must deploy fts-solr or fts-lucene, which maintain separate inverted indexes. This adds operational complexity (Solr/Lucene cluster, schema management) and storage overhead, but it is the only way to get SEARCH TEXT performance at scale.

Actionable closing

1. Inspect your current cache fields with doveadm dump index on a representative mailbox.
2. Adjust mail_cache_fields and mail_always_cache_fields to match the FETCH profile of your clients.
3. Verify index size with du -sh /var/vmail/*/*/Maildir/dovecot.index* — aim for ~100–300 bytes per message.
4. If you run on NFS, move indexes to a local filesystem and confirm ownership with ls -la /var/indexes.
5. Schedule a test doveadm force-resync on a non‑critical mailbox to validate corruption recovery time.

Following these steps gives you a measurable handle on IMAP latency, storage consumption, and recovery readiness — without guessing what the index files are doing.