Deploying Splunk Search Head Clustering for 2–3 Peers: Design, Checks, and Failure Conditions

Problem Statement

In small environments, a single Splunk Search Head can become a bottleneck and a single point of failure. Search Head Clustering (SHC) provides stateful replication of user sessions, bookmarks, and scheduled searches across peers, ensuring continuity when a head goes down. This article explains how to design the smallest viable SHC (two or three peers), where to place trust boundaries, how to validate the deployment, what failure modes to watch for, and when the design should be reconsidered.

Requirements

• Splunk Enterprise 8.x or newer (free edition does not support SHC).
• At least two Search Head instances, each with the same SPLUNK_HOME and identical splunkd.conf and web.conf.
• One Cluster Master (CM) that coordinates configuration and search allocation. The CM can be a dedicated host or co‑located with an indexer cluster master.
• Shared configuration store – typically the CM’s $SPLUNK_HOME/etc/system/local or an external network‑shared filesystem.
• Optional: Indexer clustering enabled on the CM to provide HA for the cluster master itself.
• Network connectivity: All peers must reach the CM on port 8089 (management) and the CM must reach each peer on the same port.
• Load balancing (internal round‑robin or external reverse proxy) to distribute web traffic across peers.

Minimal Design

1. Cluster Master: Install Splunk Enterprise on a host, enable SHC by adding the shcluster stanza to splunkd.conf.

   [shcluster]
   cluster_label = small_cluster
   master_uri = https://cm-host:8089
   replication_factor = 2
   

   Place this file in $SPLUNK_HOME/etc/system/local on the CM. The replication_factor matches the number of peers.

2. Search Heads: On each peer, add the same stanza to splunkd.conf.

   [shcluster]
   cluster_label = small_cluster
   master_uri = https://cm-host:8089
   replication_factor = 2
   

   Ensure the cluster_label matches the CM exactly.

3. Shared Configuration: The CM’s etc/system/local is the source of truth. Do not manually edit peer config files unless you also update the CM.
4. Load Balancer: Configure the Splunk web server with http_port = 8000 on each peer and set up a simple round‑robin reverse proxy (e.g., HAProxy) that forwards requests to peer1:8000 and peer2:8000 (and peer3:8000 if present).

Trust & Data Boundaries

SHC assumes that all peers trust the CM to provide consistent configuration. The CM holds the definitive searchhead.conf and bookmark.conf files. Peers sync these files via the CM’s cluster_master API. The data (indexes) themselves are not replicated by SHC; they remain on the indexers. Therefore, the trust boundary is limited to configuration and search state, not raw event data.

Operational Checks

1. Cluster Status – Run on every peer:

   splunk show shcluster-status
   

   All peers should report the same cluster_master and a status of active. If any peer shows read‑only, the CM is unreachable or misconfigured.

2. Replication Verification – Verify that scheduled searches and bookmarks are present on all peers.

   splunk list scheduledsearches
   splunk list bookmarks
   

   Run the commands on each peer and compare the output. Missing items indicate replication failure.

3. Log Inspection – Search $SPLUNK_HOME/var/log/splunk/splunkd.log for shcluster events.

   grep -i shcluster $SPLUNK_HOME/var/log/splunk/splunkd.log | tail -n 20
   

   Look for replication and state sync messages. Errors or warnings suggest configuration drift.

4. Fail‑over Test – Gracefully stop the CM and observe peer behavior.

   splunk stop
   

   Peers should transition to read‑only until the CM resumes. Verify that searches continue to run but new scheduled searches cannot be created.

5. Health Check – Use Splunk’s internal health endpoint:

   curl -k https://peer1:8089/services/monitor/cluster/shcluster?output_mode=json
   

   Check the cluster_status field. A value of active confirms healthy operation.

Failure Modes & Conditions

• Cluster Master Down – Without HA, the CM becomes a single point of failure. Peers go read‑only, scheduled searches cannot be modified, and new users cannot log in. Remedy: enable indexer clustering on the CM or deploy a second CM with the same cluster_label and a shared shcluster.conf file.
• Split‑Brain – In a two‑peer cluster, if the CM loses connectivity to one peer, the remaining peer may still be considered active but loses quorum. Splunk resolves this by placing the cluster in read‑only mode to prevent data inconsistency. Mitigation: add a third peer to provide quorum.
• Configuration Drift – Manually editing a peer’s config file without updating the CM will cause the CM to push the correct version later, overwriting changes. Keep all configuration changes in the CM’s shcluster.conf.
• Load Balancer Failure – If the reverse proxy goes down, users may only reach the last configured peer. Ensure the load balancer has HA or fallback rules.
• Network Partition – A split between the CM and a subset of peers can isolate those peers. They will become read‑only until connectivity is restored.

When to Reconsider the Design

1. More Than 3 Peers – For production environments with higher query volumes, add additional peers and a dedicated load balancer. The CM still coordinates but the cluster can scale horizontally.
2. High Availability Requirements – If the CM cannot tolerate downtime, enable HA by running the CM as part of an indexer cluster or by deploying a second CM with a shared shcluster.conf and a replicated filesystem.
3. Security Isolation – If you need separate user sessions or bookmarks per environment, deploy separate SHCs with distinct cluster_label values.
4. Data Residency Constraints – When search heads must reside in different geographic regions, ensure the CM’s shared config store is accessible across those regions with low latency.
5. Operational Complexity – If managing the CM and peers becomes unwieldy, consider Splunk Cloud’s managed SHC or a single Search Head with a robust load balancer and backup strategy.

Limitations & Practical Checks

• SHC is not supported in Splunk’s free edition.
• The CM’s configuration file must be kept in sync; manual edits on peers will be overwritten.
• Two‑peer clusters are vulnerable to split‑brain; they are acceptable only for low‑risk environments.
• Verify that the CM’s shcluster.conf includes replication_factor = 2 (or 3) to match the number of peers.
• After any change, run splunk show shcluster-status on all peers to confirm consistency.

By following this architecture note, you can deploy a small yet resilient Search Head Cluster, validate its operation, and know when the design needs scaling or hardening.