Run Incident Playbooks in Mattermost Without Blocking the Job Queue

Incident runbooks that sit in a wiki are useless during an outage

When an alert fires, the on-call engineer needs to know who to notify, which channel to use, and what to check next. If that information lives in a Confluence page or a Google Doc, someone has to find it, read it, and then manually copy steps into chat. By the time the runbook is open, the incident clock is already running.

Mattermost Playbooks address this by turning a runbook into an executable workflow inside Mattermost. A playbook is a template with stages, tasks, assignees, and optional webhook actions. A playbook run is one executed instance of that template. Each run gets its own channel, and the server writes every message, task change, and webhook event to a timeline that can later be exported as a retrospective.

Pick a trigger that matches how incidents start

Playbooks support three trigger types:

• Manual: started with /playbook run <name> in a channel. Best when a human declares the incident.
• Scheduled: cron-style recurrence on the server. Best for drills or periodic readiness checks.
• Webhook-driven: an HTTP POST with a JSON payload. Best for ChatOps automation from PagerDuty, Datadog, or a custom monitoring system.

The trigger determines who or what can start the response. A webhook trigger lets external systems launch a playbook without human intervention, but it also means a misconfigured payload or slow external endpoint can affect the server’s job queue.

Trigger comparison

Trigger	Starts when	Best for
Manual	/playbook run <name>	Human-declared incidents
Scheduled	Server cron schedule	Recurring checks and drills
Webhook	Incoming HTTP POST	PagerDuty, Datadog, custom alerts

Worked example: launch a playbook from a monitoring alert

Suppose your monitoring system fires a PagerDuty-like alert and you want Mattermost to create an incident channel, notify the on-call team, and post a summary to an external status page. You can configure a playbook with a webhook trigger and a task that calls a webhook action.

First, create the playbook in the Mattermost UI under Settings > Playbooks, add a webhook trigger, and copy the generated webhook URL. Then send a test payload from a terminal or CI runner:

curl -X POST '' \\
  -H 'Content-Type: application/json' \\
  -d '{\"severity\":\"critical\",\"service\":\"auth-api\",\"summary\":\"Login failures spiking\"}'

Where to run: any workstation or CI runner with network access to the Mattermost webhook endpoint. Permissions: no special Mattermost role is required to send the HTTP request, but the playbook must exist and have a webhook trigger enabled. Expected check: the server returns a success status and a new playbook run channel appears in Mattermost. Risk: if the playbook includes a webhook action to a slow external system, that call runs synchronously in the Mattermost job queue and can delay later tasks.

You can also start the same playbook manually by typing /playbook run 'Auth API Outage' in a channel where you have permission to run playbooks. A new channel should appear, and the timeline should begin recording messages and task updates immediately.

Limitations and pre-flight checks

Playbooks are powerful, but several behaviors can surprise teams that assume they work like fully asynchronous automation:

• Synchronous webhook actions: webhook tasks run in the server job queue. A flaky Jira or GitHub endpoint can block subsequent tasks until it times out. Use async middleware or a circuit breaker in front of external systems.
• Timezone drift: scheduled triggers use the server’s system timezone. In multi-region high-availability clusters, all nodes must share identical timezone data.
• Retrospective scope: export covers only the playbook run channel. Discussions in side channels such as an executive update channel are not included.
• Guest accounts: guests cannot be playbook assignees, though they can participate if added to the run channel.
• Plugin-dependent silent failures: Jira, GitHub, and Zoom actions require the corresponding plugin installed and configured. A missing plugin can cause a task to fail without an obvious error.

Before relying on playbooks in production, verify the following:

1. Create a test playbook with a webhook task pointing to a request-inspection endpoint. Trigger it with curl and confirm the payload arrives.
2. Run /playbook run <test-name> in a test channel and confirm the timeline records the run.
3. Export the retrospective and confirm it contains timestamps, decisions, and action items.
4. Check job queue health with GET /api/v4/jobs?status=pending using a System Admin token. Watch queue depth during a load test of concurrent runs.

Reverse a test playbook

Creating a playbook changes server state. To clean up after testing, delete the test playbook from Settings > Playbooks and archive the run channel it created. This removes the template and its associated channel, though exported retrospectives already downloaded remain on your local system.

Start small, then automate

Begin with one manual playbook for your most common incident type. Add a webhook trigger only after you confirm the external endpoint responds quickly and the job queue stays healthy. Re-test playbook runs after any major Mattermost upgrade, since self-hosted schema changes can affect stored definitions.