Mattermost High‑Availability Clustering: Minimal Design, Trust Boundaries, and Operational Checks

Requirements

To run Mattermost in production with high availability (HA), the system must satisfy the following core requirements:

• Stateless application tier: Any app server can serve any user request, so that horizontal scaling is straightforward.
• Shared session state: Session data must be stored in a central store (PostgreSQL or Redis) so that no server‑specific cookie or cache is required.
• Real‑time messaging: WebSocket connections and pub/sub notifications need to be delivered across all nodes.
• Fail‑over resilience: The loss of a single app server or a transient network partition should not cause user data loss or service interruption.
• Secure communication: TLS must protect client traffic to the load balancer, and internal traffic between services should be isolated by network policies.

Minimal Suitable Design

The simplest HA topology that meets the above requirements consists of:

• Two or more mattermost‑app servers behind a TCP/Layer‑7 load balancer.
• A single PostgreSQL instance (or a PostgreSQL cluster in read‑replica mode for heavy writes).
• One Redis server for pub/sub messaging.

Because Mattermost’s architecture is stateless, the load balancer can use round‑robin or least‑connection routing. Sticky sessions are discouraged; instead, the SessionStore in config.json points to PostgreSQL, and WebSocket authentication is performed against the database.

{
  "ServiceSettings": {
    "SiteURL": "https://mm.example.com",
    "ListenAddress": ":8065",
    "EnableWebSocket": true,
    "EnableOAuthService": false
  },
  "SqlSettings": {
    "DriverName": "postgres",
    "DataSource": "postgres://mm_user:mm_pass@db:5432/mm?sslmode=disable"
  },
  "RedisSettings": {
    "Host": "redis:6379",
    "Password": "",
    "DB": 0
  },
  "SessionStore": {
    "StoreType": "database"
  }
}

Deploy the load balancer (NGINX, HAProxy, or a cloud LB) to terminate TLS on mm.example.com and forward traffic to the app servers on port 8065. The LB should expose a health‑check endpoint that proxies /api/v4/system/ping to each backend.

Trust & Data Boundaries

Trust boundaries are defined at three layers:

• Client–LB: TLS terminates at the LB; all inbound traffic is encrypted.
• LB–App: Traffic is clear‑text inside the data center. Network policies should restrict this path to the LB’s IP range only.
• App–DB/Redis: Connections use TLS if enabled. The app trusts only the database and Redis; all data from clients is treated as untrusted and validated on ingestion.

Example network policy (Kubernetes):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-mattermost-db
spec:
  podSelector:
    matchLabels:
      app: mattermost-db
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: mattermost-app
    ports:
    - protocol: TCP
      port: 5432

Operational Checks

Continuous monitoring should include:

• Health‑endpoints: curl -s http://lb:8065/api/v4/system/ping should return {"status":"OK"} for each backend.
• Database latency: Run SELECT pg_sleep(0.1); from a monitoring pod and measure round‑trip time. A latency >200 ms triggers an alert.
• Redis pub/sub lag: Subscribe to the mm_user:1 channel and publish a test message. Verify that the message arrives within 50 ms. Use redis-cli --latency for automated checks.
• WebSocket connection count: Query /api/v4/users/me/active_sessions or parse the mattermost.log for WebSocket connections. A sudden drop on one node may indicate a network partition.
• Resource usage: Alert on CPU >80%, memory >70%, or disk I/O >90% for any app server.

Failure Modes

Understanding how the system behaves under failure helps design effective alerts and recovery procedures.

• App server loss: The LB simply stops routing to the failed node. Existing WebSocket connections remain open; new requests are served by other nodes. Because sessions are stored in PostgreSQL, no data is lost.
• Database outage: All nodes become unavailable for data‑dependent operations. The LB can still serve static assets, but REST and WebSocket APIs fail. A fail‑over PostgreSQL cluster or a hot standby should be in place to reduce downtime.
• Redis failure: Pub/sub messages (typing indicators, real‑time notifications) stop propagating, but the REST API and data persistence remain functional. Users may experience delayed or missing real‑time cues.
• Load balancer failure: If the LB fails, traffic stops reaching the app tier. High‑availability LB solutions (e.g., dual LB with VRRP) mitigate this risk.

Design‑Change Triggers

Certain architectural changes force a re‑evaluation of the minimal design:

• Multi‑master PostgreSQL: Requires careful transaction isolation and may introduce write conflicts. Statelessness remains, but the database connection pool must be tuned for higher concurrency.
• Kafka‑based event bus: Replacing Redis with Kafka changes the pub/sub model. The app servers must subscribe to Kafka topics, and the LB may need to expose a different health‑check path.
• End‑to‑end encryption (E2EE): Encrypting user messages on the client side means the server cannot validate message integrity. The trust boundary between client and server expands; additional key‑management services are required.
• Zero‑downtime upgrades: Rolling updates of app servers demand that the load balancer support graceful draining. The design must include health‑check timeouts and a blue‑green deployment strategy.

Practical Verification Checklist

1. Deploy two Mattermost app pods, one PostgreSQL pod, and one Redis pod in a test namespace.
2. Run curl -s http://lb:8065/api/v4/system/ping from a client pod; expect {"status":"OK"}.
3. Post a message from client A on node 1; confirm it appears instantly on client B connected to node 2.
4. Stop Redis; verify that WebSocket notifications cease but REST API calls still succeed.
5. Stop one app pod; confirm that the LB redirects new requests to the remaining pod and that existing WebSocket connections stay alive.
6. Monitor metrics in Prometheus; set alert thresholds as described above.