Architecting Real-Time State Sync in Mattermost via WebSockets

The Problem: Avoiding Database Polling at Scale

In a collaborative messaging environment, clients must receive updates—new messages, typing indicators, and reaction changes—instantly. Relying on traditional HTTP polling (where the client asks the server for updates every few seconds) creates an unsustainable load on the database and introduces unacceptable latency for the end user.

The solution is a persistent bidirectional communication channel. Mattermost implements this using WebSockets, allowing the server to push state changes to the client the moment they occur in the backend.

The Minimal Design for Real-Time Sync

The architecture relies on three primary components to move a message from a sender to a receiver without the receiver requesting it:

• WebSocket Server: Maintains a long-lived TCP connection with each active client. This removes the overhead of repeated HTTP headers for every single update.
• Pub/Sub Mechanism: A distribution layer that ensures a message sent to Server A is broadcast to a user connected to Server B. This is typically handled via the database or a dedicated message broker.
• Client-Side Sequence Tracking: A mechanism to ensure no messages are missed during brief network interruptions.

The Connection Lifecycle

The process begins with an HTTP request. The client requests a protocol upgrade to wss:// (WebSocket Secure). The server validates the session token during this handshake. Once validated, the connection is "upgraded," and the HTTP request ends, leaving a persistent socket open.

Trust and Data Boundaries

Because WebSockets bypass the standard request-response cycle of REST APIs, security must be enforced at the handshake boundary. The server does not allow a WebSocket upgrade unless a valid session token is present in the request headers or cookies.

Once the connection is established, the server maintains a mapping of the SessionID to the specific SocketID. Data boundaries are enforced by the pub/sub layer: a client only receives messages for channels they are authorized to view. The server filters the broadcast stream before pushing data down the socket, ensuring that unauthorized users cannot "listen in" on private channels by simply being connected to the server.

Operational Constraints and Scaling

Scaling WebSockets is fundamentally different from scaling stateless REST APIs. Because the server must hold the connection open in memory, resources are consumed linearly per user.

Constraint	Impact	Mitigation
Memory	Each open socket consumes server RAM.	Vertical scaling of RAM or increasing the number of server nodes.
Load Balancing	Standard round-robin can break persistent connections.	Use a load balancer that supports sticky sessions (session affinity).
Connection Limits	OS-level file descriptor limits can block new users.	Increase ulimit on the Linux host to allow more open files.

Failure Modes and Recovery

Network instability is the primary failure mode. When a WebSocket connection drops, the client enters a reconnection loop. To prevent data loss during this gap, Mattermost does not rely solely on the push notification.

The Recovery Flow:

1. The client detects a disconnection.
2. The client attempts to reconnect to the server.
3. Upon reconnection, the client sends the last received message sequence ID.
4. The server identifies the gap between the last received ID and the current head of the channel and pushes the missing messages.

Verification and Diagnostics

To verify that the real-time synchronization is functioning and not falling back to polling, you can use browser developer tools:

1. Open the Network tab in your browser.
2. Filter by WS (WebSockets).
3. Refresh the page and look for a request with a status code 101 Switching Protocols.
4. Click the request and select the Messages (or Frames) tab. You should see incoming JSON payloads in real-time as you interact with the chat.

Server-Side Checks

Administrators can monitor the health of the WebSocket layer by reviewing server logs for frequent connection_closed or handshake_failed events, which may indicate a misconfigured load balancer or an exhausted file descriptor limit.

Conditions for Design Change

This architecture is sufficient for most enterprise deployments. However, a shift in design would be required if:

• Extreme Scale: If the number of concurrent users exceeds the memory capacity of a reasonably sized cluster, the pub/sub layer would need to move from a database-backed system to a high-throughput distributed broker like Redis or NATS.
• Battery Optimization: For mobile clients, maintaining a persistent WebSocket is power-intensive. In these cases, the architecture shifts to Push Notifications (via FCM or APNs), where the WebSocket is closed and the OS wakes the app only when a notification arrives.