Configuring Scheduled Saved Searches and Alert Actions in Splunk Enterprise

You save a search, set a schedule, and expect an alert when results appear. Instead, nothing happens. The search may never have executed, or the alert action was suppressed before delivery. Before rewriting your SPL (Search Processing Language), verify two things that commonly fail outside the search interface: the scheduling role capabilities and the availability of data on your indexers.

Desired Outcome

Create a reusable saved search that runs on a fixed schedule, evaluates its results against a trigger condition, and executes an alert action—such as email, webhook POST, script execution, or index forwarding—when that condition is met. You also need a reliable way to confirm that the search ran and that the action fired.

Prerequisites and Version Assumptions

This procedure applies to Splunk Enterprise 7.x and later. Splunk Cloud Platform behaves differently in version 9.x, where alert management is separated into the Observability Cloud console with distinct permission models and API endpoints.

Platform	Alert Management Location	Permission Model	Action Constraints
Splunk Enterprise 7.x–8.x	Search & Reporting app	Role capabilities in authorize.conf	Email, script, webhook, and index forwarding available based on role and app configuration
Splunk Cloud Platform 9.x	Observability Cloud console	Tenant-level policies	Webhook and script actions often require custom add-ons or Professional Services engagement

• Role capabilities: The user creating the schedule needs can_schedule_search and can_edit. A missing can_schedule_search capability prevents scheduling silently; Splunk Web may not display an error.
• Search head access: The search head must reach the target indexers or data sources. In an indexer cluster, replication lag can make recent data unsearchable when the saved search runs.
• License: A valid Splunk license tier that supports scheduled searches.

Focused Procedure

1. Define the search and trigger

In Splunk Web, open Search & Reporting and run your SPL. For example, to detect repeated authentication failures:

index=security EventCode=4625
| stats count by src_ip
| where count > 10

Save the search as a report or saved search. In the schedule settings, choose an interval—such as every 5 minutes—and a relative time range that matches your detection window.

2. Configure the alert condition and action

Under the alert settings, set the trigger condition. A common choice is Number of Results greater than zero, but you can also use a custom condition based on field values. Add an alert action:

• Send Email: Requires configured SMTP settings in the Splunk instance.
• Webhook: POSTs results to an external URL. In Splunk Cloud, this may require a custom add-on or Professional Services engagement due to tenant-level policies.
• Run Script: Executes a script on the search head. Also often restricted in Splunk Cloud.
• Index Forwarding: Writes results to another index for downstream processing.

Set a throttling (suppression) period to prevent duplicate alerts. For example, suppress for 1 hour after the first trigger. Assign a severity if your workflow uses it for routing.

3. Save and trigger a manual run

Save the configuration. To test without waiting for the schedule, use the Run or Manual Run option on the saved search. This executes the search immediately and evaluates the alert condition.

Expected Checks and Verification

After a manual or scheduled run, verify that Splunk registered the alert and attempted delivery.

Check the Alert History dashboard

Navigate to the Alert History dashboard (accessible from the Search & Reporting app or Settings > Searches, Reports, and Alerts). Confirm that your search name appears with a recent timestamp and that the configured action is listed as triggered.

Query the REST API for active alerts

From a management host or search head with network access, list alerts to confirm registration. Replace <user>, <pass>, <host>, and <mPort> with your management credentials and port (default 8089):

curl -k -u <user>:<pass> https://<host>:<mPort>/services/data/alerts | grep -i 'search_name'

Expected check: Your saved search name appears in the output. If it is absent, the alert was not created or is disabled.

Inspect internal logs for delivery errors

Search the internal index to see whether Splunk logged errors during action execution:

index=_internal source=*metrics.log alert*

Filter by your search name if the volume is high. Look for entries indicating throttling, SMTP failures, or webhook timeouts.

Audit role capabilities on the search head

If the search never runs, inspect the effective capabilities for the role. On the search head, with filesystem access, run:

splunk btool authorize list --debug | grep can_schedule_search

Risk: This command reads configuration files and requires appropriate OS permissions. If the role lacks the capability, add can_schedule_search to role.capabilities in authorize.conf within the correct app context, then reload authentication or restart the search head.

Recovery Options and Common Failure Modes

Silent scheduling failure

A missing can_schedule_search capability prevents the search from scheduling, but Splunk Web may not surface the error. Recovery: Audit authorize.conf using btool as shown above, correct the role, and verify the schedule appears in Settings > Searches, Reports, and Alerts.

Alert suppression from indexer cluster lag

In multi-site clusters with asynchronous replication, data may not be searchable when the saved search executes. If the replication lag exceeds the throttling window, alerts are suppressed or delayed. Recovery: Extend the search time range, lengthen the throttling period, or monitor replication factor and search factor status in the Monitoring Console before concluding the SPL is wrong.

Cloud tenant policy restrictions

In Splunk Cloud Platform 9.x, webhook and script actions are constrained by tenant-level policies. If these actions fail, they may require a custom add-on deployment or Professional Services engagement. Recovery: Use native supported actions such as email, or submit a support request to review tenant policies.

Throttling hides subsequent events

If alerts stop after the first notification, the suppression period may still be active. Recovery: Reduce the throttle window if your operational need requires more frequent notification, or use severity-based suppression.

Limitations

• Platform differences: Splunk Cloud Platform 9.x separates alert management into the Observability Cloud console. The Splunk Web procedure above applies to Splunk Enterprise; Cloud users must use the console-specific permissions and API endpoints.
• No delivery guarantee: Alert actions are best-effort. Email can be filtered, and webhooks can time out. The _internal index and Alert History are the authoritative record of whether Splunk attempted delivery, not whether the recipient received it.
• Search head dependency: If the search head is unavailable, scheduled runs are missed. There is no built-in catch-up for the interval in standard saved searches.

Practical Result Check

To confirm the configuration works end to end, force a triggering condition or wait for the next scheduled interval. Then verify all four points:

1. The search appears in Settings > Searches, Reports, and Alerts with an active schedule.
2. The Alert History dashboard shows a recent run for that search.
3. The action endpoint—inbox, webhook receiver, or script log—shows the expected payload.
4. index=_internal source=*metrics.log alert* shows no errors for that search name.

If all four pass, the saved search is scheduling, evaluating, and delivering correctly. If any fail, use the recovery options above to isolate whether the issue is role configuration, data availability, or action delivery.