Splunk Ingestion Decision Guide: Choosing Between HTTP Event Collector and Universal Forwarder

Decision Context

When you need to ingest data into Splunk, you typically have two supported pathways: the HTTP Event Collector (HEC) and the Universal Forwarder (UF). Both are fully supported by Splunk, but they serve different operational scenarios. This guide helps you decide which method best fits your environment based on data type, volume, and operational overhead.

Constraints

• Data Source: Application APIs, microservices, or real‑time streams vs. system logs, archives, or binary files.
• Network Exposure: HEC requires an open TCP port (default 8088) reachable from the source; UF runs locally on the source machine.
• Both methods need indexer capacity planning to handle the incoming event rate.
• Security: HEC needs TLS and authentication tokens; UF requires agent installation and OS‑level access.

Option Comparison

Feature	HEC	UF
Agent Requirement	None – data sent over HTTP/HTTPS	Agent installed on each source host
Data Granularity	Event‑level, JSON‑friendly	File‑level, supports binary and large files
Latency	Low – real‑time push	Higher – depends on file rotation and polling
Scalability	Easy horizontal scaling via multiple tokens	Scales by adding more forwarders; requires agent management
Security	Requires TLS certificates and token authentication	Agent uses Splunk credentials; can be hardened with SSL
Operational Overhead	Minimal – configure token and endpoint	Requires installation, configuration, and ongoing maintenance
Use Case Fit	High‑volume, lightweight, structured events (e.g., REST API logs)	Bulk, file‑based ingestion (e.g., /var/log, Windows Event Logs)

Trade‑Offs

• HEC offers simplicity and near‑real‑time ingestion but is limited to structured event payloads. It also exposes a port that must be secured.
• UF provides comprehensive file monitoring, including binary data and large archives, but introduces agent overhead and slower ingestion for highly dynamic data.
• For environments where both structured API logs and system logs exist, a hybrid approach—HEC for application logs and UF for system logs—often yields the best overall coverage.

Concrete Implementation Example

1. Configure HEC on the Indexer

Run the following on the indexer (requires splunk user privileges):

splunk enable web
splunk restart

Navigate to https://<indexer-host>:8089/settings/data_inputs/http_event_collector and enable the HEC port (default 8088). Create a token named app_logs_token with the following settings:

• Index: main
• Source type: json
• Enable SSL: Yes (upload your CA cert)

After creation, note the token value; it will be used in the client.

2. Send a Sample Event via cURL

Run this command from an application host or a test machine. Replace placeholders accordingly.

curl -k \
  -H "Authorization: Splunk <HEC_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"event": {"app": "orders", "status": "created", "user_id": 12345}}' \
  https://<indexer-host>:8088/services/collector

Expected: HTTP 200 OK and the event appears in the main index.

3. Configure UF on a Log Source

Install the UF on the source machine (Linux example):

wget https://download.splunk.com/products/universalforwarder/releases/8.2.1/linux/splunkforwarder-8.2.1-xxxx-Linux-x86_64.tgz
sudo tar -xzf splunkforwarder-8.2.1-xxxx-Linux-x86_64.tgz -C /opt
sudo /opt/splunkforwarder/bin/splunk start --accept-license

Create inputs.conf in /opt/splunkforwarder/etc/system/local/ to monitor all log files in /var/log:

[monitor:///var/log/*.log]
index = main
sourcetype = syslog

Configure outputs to forward to the indexer:

[default]
server = <indexer-host>:9997
sslCertPath = /opt/splunkforwarder/etc/auth/mycert.pem
sslPassword = <cert_password>

Restart the UF:

sudo /opt/splunkforwarder/bin/splunk restart

4. Verify Both Paths

Run a search on the indexer to confirm events from both sources:

index=_internal sourcetype=splunkd_access | head 10

Look for entries with source=hec (HEC) and source=forwarder (UF). You can also search the specific index:

index=main sourcetype=json OR sourcetype=syslog | head 10

Limitations & Practical Checks

• HEC requires a valid TLS certificate; if the cert is self‑signed, clients must trust it or use -k to skip verification (not recommended for production).
• High‑frequency HEC traffic can overwhelm an under‑provisioned indexer; monitor splunkd_access for 429 responses and scale the indexer or throttle the source.
• UF needs OS access; ensure the agent runs as a privileged user if monitoring system directories.
• Always back up configuration files before making changes; use Splunk's splunk backup command for critical components.

Conclusion

Choose HEC when you need lightweight, low‑latency ingestion of structured events from APIs or microservices. Opt for UF when you must ingest large volumes of file‑based logs, binary data, or when you prefer a managed agent that handles file rollover and compression. In many production environments, a hybrid strategy—HEC for application logs and UF for system logs—provides the most comprehensive coverage while balancing operational overhead.