Speeding Up Dovecot Mailbox Search with the Solr FTS Backend

Problem: Slow IMAP SEARCH on Large Mailboxes

When a mailbox grows to hundreds of thousands of messages, a plain IMAP SEARCH that scans each file can take several seconds, hurting user experience and increasing CPU load on the Dovecot server. The goal is to offload the heavy lifting to a dedicated search engine so that queries return matching UIDs in sub‑second time while keeping Dovecot’s own resource usage low.

Thesis: Enable Dovecot’s Full‑Text Search with the Solr Backend

By installing the dovecot-fts-solr plugin, configuring a Solr core, and letting Dovecot push each new or changed message to Solr via HTTP, you turn mailbox search into a fast query against an inverted index. Existing mail can be bulk‑indexed with doveadm fts rescan, after which IMAP SEARCH using TEXT or BODY is translated to Solr queries and returns results without touching the mailbox files.

Section 1: Prerequisites and Plugin Installation

Run these commands as root on the Dovecot host (adjust package manager for your distro):

# Install the FTS Solr plugin (example for Debian/Ubuntu)
apt-get update
apt-get install dovecot-fts-solr

Verify the plugin is available:

doveadm plugin list | grep fts_solr

Expected output (no invention): a line containing fts_solr if the plugin is correctly installed.

Section 2: Configuring Dovecot and Solr

Edit /etc/dovecot/conf.d/10-mail.conf (or a local override) to add the FTS plugins:

mail_plugins = $mail_plugins fts fts_solr

Create a Solr core for mail (assume Solr is already running). Example using the Solr CLI:

solr create -c mailcore -n data_driven_schema_configs

Then point Dovecot to that core in /etc/dovecot/conf.d/20-imap.conf (or a dedicated FTS file):

plugin {
  fts = solr
  fts_solr_url = http://:/solr/
  fts_solr_user =    # optional, if Solr requires auth
}

Replace placeholders:

• – hostname or IP of the Solr server
• – usually 8983
• – the core you created, e.g., mailcore
• – username for basic auth, if needed

After saving, restart Dovecot:

systemctl restart dovecot

Section 3: Indexing Existing Mail and Verification

To index all current messages, run doveadm fts rescan as the dovecot user (or a user with read access to mail storage):

sudo -u dovecot doveadm fts rescan -A

This command walks through every user’s mailbox and POSTs each message to Solr. Expect increased network and disk I/O; schedule it during off‑peak hours.

Verification steps (run as root or with sudo):

1. Check plugin load: doveadm plugin list | grep fts_solr
2. Confirm a document appears in Solr: after delivering a test message, open Solr’s admin UI (http://:/solr//browse) and query for the message‑ID; you should see one document.
3. Test a search: doveadm search TEXT "hello" returns the UID(s) of messages containing “hello”.
4. Compare latency: run the same search with the sequential scan optimizer disabled: doveadm search -S all TEXT "hello" and note the time difference (use time prefix if desired). The FTS version should be markedly faster.

Trade‑offs and Limitations

While search latency improves, you now operate an extra service:

• Operational complexity: Solr requires JVM tuning, heap monitoring, and periodic core reloads when you change the schema.
• Initial indexing cost: Bulk‑indexing large mailboxes can saturate network and I/O; mitigate with incremental rescans (doveadm fts rescan -u user@example.com) or by throttling the POST rate via Solr’s request handler.
• Schema changes: Adding new fields to the Solr schema necessitates a core reload and a full reindex to make the fields searchable.

Practical way to check that the trade‑off is acceptable: monitor Solr’s query latency (qt=standard in the admin UI) and Dovecot’s CPU usage (top or pidstat) during normal search traffic. If Solr stays under 50 ms average query time and Dovecot CPU drops compared to sequential scans, the setup is delivering the intended benefit.

Actionable Closing

If your users complain about slow mailbox search and you can spare a modest Solr instance, follow the steps above to enable the FTS Solr backend. Start with a test user, verify indexing and query speed, then roll out to all accounts. Keep an eye on Solr heap and Dovecot I/O during the initial rescan, and schedule that work outside peak hours to avoid service impact.