Mattermost Plugin Architecture: Trust Boundaries and Operational Safety

The Problem: Extending Core Behavior Without Forking

Teams running Mattermost often need custom bots, slash commands, UI modifications, or webhook integrations. The naive approach—modifying server source code—creates an upgrade trap: every upstream release requires re-applying patches, and security fixes stall behind merge conflicts. Mattermost's plugin system solves this by providing a supported extension surface that keeps the core server upgradeable while isolating third-party code.

Requirements Driving the Design

• Upgrade independence: Core server updates must not require plugin recompilation or reinstallation beyond documented API version bumps.
• Auditability: Administrators must verify what code runs, what data it accesses, and whether it originates from a trusted source.
• Operational containment: A misbehaving plugin must not crash the main server process or block API endpoints indefinitely.
• Supported distribution: Installation via the System Console and Marketplace, with signature verification, not manual binary drops.

Smallest Suitable Design: Mediated RPC Over Process Isolation

On server start, the Mattermost core reads each plugin's plugin.json manifest, which declares the plugin's ID, version, required server API version, and entry points. For Go-backed plugins (the only backend type currently supported), the server spawns a separate child process per plugin. Communication occurs over a gRPC channel using the plugin SDK—a generated client/server pair that defines every permissible call.

The plugin process has no direct database handle, no filesystem access outside its designated bundle directory, and no raw HTTP listener. All data access—reading posts, creating users, storing key-value pairs—routes through the SDK's RPC methods, which the core server implements with its own authorization checks. Frontend assets (React bundles, CSS) are served by the core's static file handler under /plugins/{id}/static/ after the plugin registers its bundle hash.

Concrete example: A plugin implementing a Jira webhook declares in its manifest:
"backend": { "executable": "plugin-linux-amd64" }, "api_version": "v1"
At runtime, the core launches plugin-linux-amd64 as a child process, establishes the gRPC link, and exposes the plugin's HTTP endpoints under /plugins/jira/webhook via the core's router. The plugin calls api.KVSet() to store OAuth tokens; the core persists them in its own KV store with the plugin's ID as namespace prefix.

Trust and Data Boundaries

The plugin runs in the same OS user context as the Mattermost server (typically mattermost), sharing the same network namespace and file descriptor table. Isolation is logical, not containerized: the SDK mediates every privileged operation. This means:

• Credentials: Database passwords, JWT secrets, and config values are not injected into the plugin process. The plugin requests operations (e.g., GetPost, CreateUser) and the core enforces permissions.
• File access: Plugins read/write only within ./plugins/{id}/ via api.GetBundlePath() and api.WriteFile(). Arbitrary filesystem reads are impossible without a server-side vulnerability.
• Network: Outbound HTTP calls from plugins use the core's api.HTTPClient(), which respects the server's proxy and TLS settings. A plugin cannot open raw sockets or bind ports.

Limitation: A compromised plugin inherits the server's OS privileges. If the Mattermost process can read /etc/shadow (it shouldn't), so can the plugin. Treat the plugin boundary as a software contract, not a security sandbox.

Operational Checks and Observability

The System Console (System Console → Plugins → Management) surfaces:

• Backend status: Running, Stopped, or Error with the plugin process's last exit code.
• Structured logs: Plugin stdout/stderr interleaved with core logs, tagged by plugin ID. Filter with journalctl -u mattermost -g "plugin_id=jira" on systemd hosts.
• Resource usage: CPU and memory per plugin process (requires cgroup v2 or manual ps correlation).
• Enable/disable toggle: Stops the child process and deregisters routes without core restart.
• Signature verification: Marketplace plugins carry a Mattermost-signed SHA256 hash; the core verifies on upload. Self-signed plugins require PluginSettings.EnableInsecureDownload (default false).

Diagnostic decision: If a plugin shows "Error" status, check its logs first. A common failure is API version mismatch—e.g., plugin built against v1.21 but server runs v1.20. The manifest's api_version" field must be ≤ server's advertised version.

Failure Modes and Containment

Failure	Containment	Operator Impact
Plugin panic (nil pointer, OOM)	Child process exits; core detects via gRPC keepalive and marks plugin Stopped	Core API remains responsive; plugin endpoints return 503
Plugin deadlock (blocking on mutex)	gRPC calls from core to plugin time out (default 5s); core logs timeout, returns 504 to caller	Elevated latency on plugin routes only; other plugins unaffected
Accidental data leakage via permitted API	Plugin calls GetPostsForChannel on private channel it shouldn't access	Core's authorization layer enforces channel membership; plugin receives 403
Manifest version skew after server upgrade	Core refuses to start plugin if api_version > server version	Plugin stays disabled; admin must update or downgrade plugin

Core server availability is preserved because the plugin process is fully optional. Disabling a plugin via the console (or mmctl plugin disable jira) sends SIGTERM to the child process and removes its routes from the router within seconds.

Conditions That Would Change This Design

The current architecture assumes plugins are cooperative extensions—they enhance workflows but don't require privileged access. The design would shift if requirements demanded:

• OS-level privileges: Loading kernel modules, manipulating iptables, or mounting filesystems. This would require a separate agent model with explicit capability grants, moving beyond the plugin SDK.
• Cross-tenant data access: A plugin needing to query messages across multiple Mattermost instances (e.g., a compliance archiver). This implies a federation layer or external service, not an in-process extension.
• Custom storage engines: Replacing PostgreSQL with a graph database for plugin data. The KV and plugin data interfaces are intentionally opaque; storage abstraction is a core server concern.
• Real-time stream processing: Sub-millisecond latency on every message event. The current gRPC hop adds ~1-2ms; a sidecar or embedded WASM runtime would be needed.

In each case, the solution moves from "plugin" to "server fork" or "enterprise feature" with its own release cadence and support contract.

Verification Checklist for Operators

1. In a test instance, open System Console → Plugins → Management. Install a Marketplace plugin (e.g., "Zoom"). Confirm backend status shows "Running" and logs show successful gRPC handshake.
2. Inspect the plugin's extracted bundle under ./plugins/zoom/. Verify plugin.json declares api_version" compatible with your server version (mmctl version).
3. Disable the plugin via the console. Confirm the child process terminates (ps aux | grep zoom shows no plugin process) and core API endpoints remain healthy (curl -f http://localhost:8065/api/v4/system/ping).
4. Re-enable and simulate a crash: kill -SEGV $(pgrep -f 'plugin-linux-amd64.*zoom'). Verify core logs record the exit and plugin status transitions to "Error" without core restart.

These steps validate that the mediation layer works as designed and that your operational tooling (monitoring, alerting on plugin health) catches the documented failure modes.