Splunk Universal Forwarder to Heavy Forwarder Pipeline Architecture: Design, Trust Boundaries, and Operational Guardrails

The Core Problem: Decoupling Ingestion from Processing at Scale

High-volume Splunk deployments hit a wall when Universal Forwarders (UFs) send raw data directly to Indexers. The Indexer tier becomes a bottleneck because it must simultaneously parse, transform, enrich, and write data—while also serving search requests. The smallest suitable design that solves this inserts a Heavy Forwarder (HF) cluster between UFs and Indexers. UFs handle lightweight, secure data collection; HFs absorb CPU-intensive parsing, masking, and routing; Indexers focus on indexing and search performance.

Smallest Suitable Design: UF → Load-Balanced HF Cluster → Indexer Tier

Deploy Universal Forwarders on data sources (servers, containers, network devices). Configure them to forward to a DNS-round-robin or load-balancer VIP fronting 3+ Heavy Forwarders. HFs run props.conf/transforms.conf for parsing, SEDCMD for PII masking, and outputs.conf to distribute events across Indexers. This three-tier topology isolates failure domains: a UF outage affects one source; an HF outage triggers UF-side buffering; an Indexer outage triggers HF-side queuing.

Concrete UF Configuration for Persistent Queuing

On each Universal Forwarder, enable persistent queuing in $SPLUNK_HOME/etc/apps//local/outputs.conf to survive HF downtime without data loss:

[tcpout]
defaultGroup = hf_cluster

[tcpout:hf_cluster]
server = hf-vip.example.com:9997
useACK = true

# Persistent queue: 2 GB on disk, 500 MB in memory
persistentQueueSize = 2GB
persistentQueueMaxFileSize = 500MB

Where to run: On every UF host. Permissions: Splunk service account write access to $SPLUNK_HOME/var/run/splunk/. Risk: Disk-full on UF host halts ingestion; monitor persistentQueueSize usage via splunk show queue or the Monitoring Console.

Trust and Data Boundaries at the Heavy Forwarder Layer

The HF tier is the trust boundary where sensitive data is transformed before it ever reaches the Indexer. This is where you enforce:

• PII masking: SEDCMD-pii = s/(\d{3}-\d{2}-\d{4})/XXX-XX-XXXX/g in props.conf for SSN redaction.
• Protocol deconstruction: Parse syslog, CEF, JSON, or custom formats using LINE_BREAKER, SHOULD_LINEMERGE, and KV_MODE.
• Routing decisions: Send security logs to index=security, debug logs to index=debug, and drop noisy events via transforms.conf REGEX = . DEST_KEY = queue FORMAT = nullQueue.

Because HFs parse and transform, they consume 3–5× more CPU per GB than UFs. Size HF nodes for peak transformation load, not just throughput. A typical starting point: 16 vCPU / 64 GB RAM per HF handling ~500 GB/day of parsed data.

Operational Checks: Metrics That Reveal Pipeline Health

Monitor two metric families to catch bottlenecks before they cause data lag:

1. HF Blocked Queue Metrics

Run this search on the Monitoring Console or a search head connected to the _introspection index:

index=_introspection component=metrics name=queue 
| where blocked=true 
| stats max(current_size_kb) as max_blocked_kb by host, name 
| eval max_blocked_mb=round(max_blocked_kb/1024,2) 
| where max_blocked_mb > 100

Alert when any HF queue (parsingQueue, typingQueue, indexQueue) stays blocked > 100 MB for 5 minutes. This indicates CPU saturation or downstream Indexer backpressure.

2. Indexer Indexing Latency

index=_internal source=*metrics.log group=indexer 
| stats avg(indexing_latency) as avg_latency_ms by host 
| where avg_latency_ms > 5000

Latency > 5 seconds means Indexers cannot keep up. Check disk I/O, bucket rollover frequency, and search concurrency.

Failure Modes and Mitigations

Failure Scenario	Impact	Mitigation
Single HF node fails	UFs redistribute to remaining HFs via useACK=true; brief latency spike	Run 3+ HFs behind VIP; configure autoLBFrequency = 30 in outputs.conf for faster failover
Entire HF cluster unreachable	UFs buffer to persistent queue until disk fills	Size persistent queue for max expected outage (e.g., 2 GB ≈ 4 hrs at 150 GB/day); alert on queue fill > 80%
Indexer cluster down	HFs queue to indexQueue; eventually block parsingQueue	Enable dropEventsOnQueueFull only for low-priority sourcetypes; monitor blocked flag on HF indexQueue
Regex transformation runaway	HF CPU 100%, parsingQueue blocks, data lag grows	Limit MAX_EVENTS in props.conf; use TRUNCATE for oversized events; profile with splunk cmd btool props list --debug

Verification Procedures

1. Data flow continuity: Compare event counts at each tier over a 15-minute window:

   # On HF: events processed
   index=_internal source=*metrics.log group=per_sourcetype_thruput 
   | stats sum(kb) as kb_processed by series 
   
   # On Indexer: events indexed
   index=_internal source=*metrics.log group=per_index_thruput 
   | stats sum(kb) as kb_indexed by series 
   | eval diff_kb = kb_processed - kb_indexed
   

   Investigate any series where diff_kb > 5% of processed volume.
2. Failure recovery test: In a maintenance window, stop one HF (splunk stop). Verify UF persistentQueue grows, then flushes when HF restarts. Check splunkd.log for Connected to idx=... messages.
3. Masking validation: Send a test event with known PII:

   logger -p local0.info "User SSN 123-45-6789 logged in"
   

   Search the Indexer: index=main "XXX-XX-XXXX". Confirm original SSN never appears in _raw.

When This Design Changes: Moving to Indexer Clustering

The UF→HF→Indexer design assumes a standalone Indexer or a non-clustered Indexer tier. If requirements shift to:

• Multi-site high availability with automatic failover
• Data replication (RF=3, SF=2) for durability
• Searchable copies across sites for low-latency search

…then you must adopt an Indexer Cluster with a Cluster Master (CM). The HF outputs.conf changes from static server= list to master_uri = https://cm.example.com:8089 and indexerDiscovery = cm. The CM coordinates bucket replication, primary assignment, and search factor compliance. This adds operational complexity: CM becomes a single point of failure (mitigate with CM redundancy), and bucket fixup traffic consumes inter-node bandwidth.

Limitations and Practical Constraints

• HF compute cost: Complex regex (e.g., multi-line Java stack traces) can exhaust CPU. Profile with splunk search 'index=_introspection component=PerProcessMetrics' | stats avg(cpu_percent) by process_name.
• Persistent queue disk: UF hosts need 2–5 GB free per 100 GB/day ingestion. Add a disk monitor: df -h $SPLUNK_HOME/var/run/splunk/persistentqueues.
• Version assumptions: Configurations tested on Splunk Enterprise 9.1+. persistentQueueMaxFileSize requires 8.2+. indexerDiscovery requires CM 8.0+.
• No encryption at rest on UF queues: Persistent queues are plaintext. If UF hosts are untrusted, encrypt the filesystem or avoid persistent queuing for sensitive sources.

Quick Diagnostic Checklist

• UF outputs.conf has useACK=true and persistentQueueSize set
• HF props.conf masks PII before TRANSFORMS-routing runs
• Monitoring Console alerts on HF blocked=true > 5 min
• Indexer indexing_latency < 5 sec sustained
• Quarterly failure-recovery drill: stop one HF, verify buffer drain