Configuring Mattermost Read Replicas: Minimal Design, Trust Boundaries, and Operational Checks

Problem: Primary Database Overload in Large Mattermost Deployments

As a Mattermost installation grows, the primary PostgreSQL (or MySQL) instance can become a bottleneck for read‑heavy workloads such as channel browsing, user searches, and API polling. Adding read replicas lets the application direct SELECT queries to replica nodes while keeping INSERT/UPDATE/DELETE traffic on the primary, reducing primary load without changing the application code.

Requirements

• Mattermost Team or Enterprise edition (read‑replica support is present in v5.30+; verify your exact version).
• A PostgreSQL or MySQL primary with at least one asynchronous replica that is caught up within an acceptable lag window (typically a few seconds).
• Network connectivity from all Mattermost app nodes to both primary and replica (same VPC/subnet or peered network).
• A database user with sufficient privileges to perform SELECT on all tables on both primary and replica (the same credentials are used for both).

Smallest Suitable Design

The minimal topology that satisfies the requirements is:

• One primary database node.
• One read replica node.
• Mattermost app nodes configured with DataSourceReplicas pointing at the replica.

No external connection‑pooler (e.g., PgBouncer) is required unless the total number of database connections from all app nodes exceeds what Mattermost’s built‑in pool can handle (default MaxIdleConns = 20, MaxOpenConns = 100 per datasource). If you anticipate >200 concurrent connections per node, consider adding a pooler; otherwise, rely on Mattermost’s internal pooling.

Configuration Example

Edit /opt/mattermost/config/config.json on each app node (requires read/write permission on the file and the ability to restart the Mattermost service).

{
  "SqlSettings": {
    "DriverName": "postgres",
    "DataSource": "postgres://mmuser:{{PRIMARY_PASSWORD}}@{{PRIMARY_HOST}}:5432/mattermost?sslmode=disable&connect_timeout=10",
    "DataSourceReplicas": [
      "postgres://mmuser:{{REPLICA_PASSWORD}}@{{REPLICA_HOST}}:5432/mattermost?sslmode=disable&connect_timeout=10"
    ],
    "MaxIdleConns": 20,
    "MaxOpenConns": 100,
    "ConnMaxLifetime": 3600
  }
}

Placeholders:

• {{PRIMARY_HOST}} – hostname or IP of the primary database.
• {{REPLICA_HOST}} – hostname or IP of the read replica.
• {{PRIMARY_PASSWORD}} and {{REPLICA_PASSWORD}} – passwords for the mmuser account (can be identical if the same role is used on both nodes).

After saving the file, restart Mattermost:

# Run as root or with sudo
systemctl restart mattermost

Trust and Data Boundaries

All Mattermost nodes share the same database credentials, so the trust boundary is at the network layer, not the application layer. To limit exposure:

• Place primary and replica in a private subnet accessible only from the Mattermost app subnet.
• Enforce TLS between app nodes and the database (set sslmode=require in the DSN if your DB supports it).
• Use the least‑privilege role: mmuser needs SELECT, INSERT, UPDATE, DELETE on the Mattermost schema; no superuser rights are required.
• Treat the replica as equally sensitive as the primary because it holds a full copy of the data; apply the same encryption‑at‑rest and backup policies.

Operational Checks

Once replicas are enabled, monitor the following:

1. Replication lag – for PostgreSQL, run SELECT now() - pg_last_xact_replay_timestamp() AS lag; on the replica; for MySQL, SHOW SLAVE STATUS\G and check Seconds_Behind_Master. Set an alert if lag exceeds 5 seconds (adjust based on your tolerance for stale reads).
2. Datasource usage – Mattermost logs (/var/log/mattermost/mattermost.log) contain lines like [SQL] SELECT … from replica when a query is routed to a replica. You can also enable the Diagnostics endpoint (/api/v0/diagnostics) and inspect the database section for replicaQueryCount vs primaryQueryCount.
3. Connection pool saturation – monitor SqlSettings.MaxOpenConns usage via the database’s pg_stat_activity (PostgreSQL) or performance_schema (MySQL). Consistently hitting the max indicates a need to increase pool size or add a pooler.
4. Error rates – watch for level=error msg="Failed to execute query" in the Mattermost logs; a spike may indicate replica unavailability or lag‑induced timeouts.

Failure Modes That Should Trigger a Design Change

• Replica loss or prolonged lag – Mattermost will continue to operate, routing reads to the primary if the replica is unreachable (the driver returns an error and the app falls back). However, if lag regularly exceeds your acceptable window, user‑perceived consistency degrades (e.g., a newly posted message may not appear in channel views for several seconds). Sustained lag >10 seconds is a signal to either improve replication throughput (upgrade replica hardware, tune max_standby_streaming_delay) or reconsider the read‑replica approach.
• Write‑heavy bursts** (bulk imports, compliance exports) – replicas do not alleviate write load; the primary remains the bottleneck. If write throughput becomes the limiting factor, consider sharding, upgrading the primary, or offloading bulk operations to a separate ETL pipeline.
• Need for automated failover** – Mattermost does not promote a replica to primary automatically. If your SLA requires automatic promotion upon primary failure, you must deploy an external orchestration layer (e.g., Patroni, PostgreSQL‑operator, or a managed service with built‑in failover) and adjust the DSN to point to a virtual IP or service name that the orchestrator updates.

Conditions That Would Prompt a Redesign

• Sustained primary CPU or I/O saturation despite replica offload – indicates that read splitting alone isn’t enough; consider adding more replicas, upgrading the primary, or moving to a read‑scale‑out architecture (e.g., Citus, Vitess).
• Multi‑region deployments where latency to the primary is high for remote users – you may need region‑specific read replicas and a global traffic manager to direct users to the nearest replica.
• Compliance or analytics requirements that demand isolated read‑only access to historical data – a dedicated reporting replica (or a change‑data‑capture pipeline to a data warehouse) may be more appropriate than sharing the same replica used for interactive reads.

Practical Verification Steps

1. Confirm version support: grep -i "DataSourceReplicas" /opt/mattermost/config/config.json should return the setting; check mattermost version against the release notes.
2. Run a load test (e.g., using mattermost-loadtest or a custom script) that simulates steady channel posting and polling. While the test runs, query replica lag and compare primaryQueryCount vs replicaQueryCount from the diagnostics endpoint.
3. Simulate replica loss: stop the replica database service or block network traffic to it. Observe Mattermost logs for fallback messages and verify that the site remains responsive (though possibly slower). Restart the replica and confirm that queries resume to it.

Limitations

• Read‑replica configuration does not improve write throughput.
• Replication lag can cause stale reads; Mattermost mitigates some cases internally but cannot guarantee strong consistency.
• Search relevance still depends on Elasticsearch/OpenSearch; replicas do not replace a dedicated search cluster.
• Edition‑specific: the feature is present in Team and Enterprise editions; the free Community edition does not include DataSourceReplicas.

By following the minimal primary‑plus‑replica design, enforcing network‑level trust boundaries, monitoring lag and query distribution, and planning for the failure modes outlined above, you can safely offload read load from your Mattermost primary database while keeping the operational complexity low.