Search head pooling concurrency limits: missing per-role cap

Direct Answers

Impact of max_searches_per_user on latency with pooling: The limit is enforced per search head, not globally across the pool. A user's total concurrent capacity scales with the number of pool members. When the pool saturates, queuing shifts between heads, causing latency spikes that are not visible under isolated load.

Interaction of max_searches_per_app with role-based limits: max_searches_per_app acts as an additional per-search-head ceiling per application, applied after per-user checks. Splunk has no native max_searches_per_role parameter, so a role's peak concurrency is bounded only by the sum of per-user and per-app limits across all pool members.

Should a per-role cap be introduced? Yes. Without it, a role with many active users can collectively consume most pooled search head slots, starving other roles even when per-user limits are respected.

Confirmed Behavior

• limits.conf defines max_searches_per_user and max_searches_per_app; both are enforced per search head.
• Search head pooling distributes incoming searches across pool members via the deployment server load balancer.
• No max_searches_per_role parameter exists in limits.conf. Role enforcement is indirect through user capabilities and per-user limits.

Likely Implications (Version-Sensitive)

• Enabling max_searches_per_user with pooling can increase perceived latency for users in a busy role because the limit applies per head; total user capacity = max_searches_per_user × pool size.
• max_searches_per_app is evaluated after per-user checks on each head, providing a second ceiling but still per head.
• Without a per-role aggregate cap, a role with many concurrent users can dominate the pool, causing queue buildup and latency spikes for other roles.

Verification Steps

1. Inspect limits.conf on a search head for max_searches_per_user and max_searches_per_app values.
2. Check server.conf and deployment server configuration to list pool members and pooling mode.
3. Review search jobs and dispatch queue metrics in the Splunk UI under concurrent load.
4. Correlate active searches by user and role during peak periods to assess whether a single role dominates pool capacity.

One Diagnostic Detail Needed

What Splunk Enterprise major version is deployed? Limits semantics and pooling implementation changed across major releases, and the exact queuing behavior under saturation is version-dependent.