Speeding up IMAP search with Dovecot’s Lucene‑based full‑text search

The problem: slow IMAP searches on large mailboxes

When a mailbox grows to hundreds of thousands of messages, a plain IMAP SEARCH forces Dovecot to scan every mail file on disk. Clients notice the delay as a lag of several seconds or more, especially on busy servers or when users search across multiple folders.

Why the Lucene FTS plugin helps

Dovecot ships with a plug‑in that offloads text search to an external Lucene index. Once the index is built, the server can answer most SEARCH TEXT or SEARCH BODY requests by consulting the index instead of reading each message. The result is typically sub‑second response times even with millions of messages.

Enabling the plugin – a step‑by‑step example

1. Install the plugin package (adjust for your distribution). On Debian/Ubuntu:

   sudo apt-get install dovecot-fts-lucene

   Run as root or with sudo.
2. Activate the plugin in Dovecot configuration. Add or edit the block in /etc/dovecot/conf.d/10-mail.conf:

   plugin {
     fts = lucene
     fts_lucene = /var/lib/dovecot/fts-lucene
   }

   The path fts_lucene is where the index files will be stored. Ensure the directory exists and is owned by the dovecot user:

   sudo mkdir -p /var/lib/dovecot/fts-lucene
   sudo chown dovecot:dovecot /var/lib/dovecot/fts-lucene

3. Restart Dovecot to load the new plug‑in:

   sudo systemctl restart dovecot

4. Build the initial index for existing mail. Run the rescan command for all users (requires dovecot user privileges):

   sudo doveadm fts rescan -A

   This walks through every mailbox and feeds the text to Lucene. Depending on mail volume, it may take minutes to hours and will increase disk I/O while it runs.
5. Verify that the plug‑in is active:

   dovecot -n | grep fts

   You should see lines showing plugin { fts = lucene } and the configured index path.
6. Check index status:

   doveadm fts status

   The output lists each user and the number of indexed documents. Non‑zero counts indicate the index contains data.
7. Test a search via IMAP. From a client or using telnet/openssl:

   openssl s_client -connect localhost:993 -quiet
   # after login:
   SEARCH TEXT "urgent"
   

   If the plug‑in is working, the server returns matching UIDs almost instantly. Compare the timing with a search on a non‑indexed field (e.g., SEARCH HEADER X‑Spam‑Flag) to feel the difference.

Trade‑offs and limitations

• Extra RAM and disk I/O: The Lucene index consumes memory proportional to the total indexed terms and generates write activity whenever new mail arrives. On very low‑end servers (e.g., < 1 GB RAM) this may noticeably affect overall performance.
• Index storage size: Expect the index to occupy roughly 20‑40 % of the raw mailbox size, depending on language and average message length. Monitor free space with df -h on the index partition.
• Version requirement: The plug‑in API is present in Dovecot 2.2.27 and later. Older releases will fail to load the plug‑in and log an error; upgrading Dovecot is required before proceeding.

Practical rollout advice

Start with a single domain or a subset of users. After enabling the plug‑in and running doveadm fts rescan for those accounts, watch the index size via doveadm fts status and monitor system load (top or vmstat). If resource usage stays within acceptable limits, expand the rescan to additional users in stages.

When you are satisfied, make the configuration permanent and consider adding a cron job or systemd timer that runs doveadm fts rescan -A nightly to keep the index up‑to‑date for new mail without impacting peak hours.

Actionable closing

If your users complain about slow mailbox searches, the Lucene‑based FTS plug‑in offers a measurable latency improvement with a straightforward setup. Verify the plug‑in loads, confirm the index builds, and test a few IMAP searches to see the response‑time drop. Keep an eye on RAM and disk usage, and scale the rollout gradually to match your server’s capacity.