Ingesting Structured Webhooks using the Logstash HTTP Input Plugin

The Problem: Handling Asynchronous Webhook Data

Many external services—such as GitHub, Stripe, or custom internal APIs—send data via HTTP POST requests (webhooks). If your data pipeline relies on polling a database or reading files, you miss the real-time nature of these events. To ingest this data directly into your ELK stack, Logstash must act as a listener rather than a collector.

The takeaway: By configuring the http input plugin with a JSON codec, Logstash transforms from a log-shipper into a lightweight API endpoint capable of parsing structured payloads into searchable fields immediately upon arrival.

Prerequisites

• Logstash Instance: A running installation (Version 7.x or 8.x recommended).
• Network Access: An open port (typically 8080) on the host firewall to allow incoming traffic from the webhook source.
• JVM Memory: If you expect payloads larger than 1MB or high concurrency, ensure the jvm.options file has sufficient heap space (e.g., -Xmx2g) to prevent OutOfMemoryError during request buffering.

Configuring the HTTP Pipeline

The following configuration sets up a listener that expects JSON data, normalizes the timestamp, and routes the event to Elasticsearch.

input {
  http {
    port => 8080
    codec => "json"
    # Limits the size of the request body to prevent memory exhaustion
    max_length_bytes => 1048576 
  }
}

filter {
  # Ensure the webhook's timestamp is used as the event time
  if [timestamp] {
    date {
      match => [ "timestamp", "ISO8601" ]
      target => "@timestamp"
    }
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "webhook-data-%{+YYYY.MM.dd}"
  }
  # Useful for debugging during initial setup
  stdout { codec => rubydebug }
}

Key Configuration Details

• codec => "json": This is critical. Without it, Logstash treats the entire POST body as a single string. With it, the JSON keys become top-level fields in the Logstash event.
• max_length_bytes**: This prevents a malicious or malformed request from crashing the JVM by limiting the maximum size of the incoming payload.
• port**: The port Logstash binds to. Ensure this does not conflict with other services on the host.

Deployment and Verification

Run the pipeline using the Logstash binary. You will need permissions to bind to the specified port (ports below 1024 usually require root/sudo).

Command:
bin/logstash -f /path/to/your/config.conf

Verification Step 1: Log Check
Check the console output or logstash-plain.log. Look for the message Pipeline started and ensure there are no BindException errors indicating the port is already in use.

Verification Step 2: Payload Test
From a remote machine or the local terminal, send a test JSON payload using curl:

curl -X POST http://<LOGSTASH_IP>:8080 
     -H "Content-Type: application/json" 
     -d '{"event": "test_webhook", "status": "success", "timestamp": "2026-10-04T12:00:00Z"}'

Expected Result: The curl command should return an HTTP 200 OK. In the Logstash stdout output, you should see the JSON keys event and status parsed as individual fields.

Performance Tuning and Security

Handling High Volume

If you observe 503 Service Unavailable errors or timeouts under load, adjust the max_workers setting in the http input block. This determines how many concurrent HTTP requests Logstash can handle. Increasing this value requires a corresponding increase in JVM heap memory.

Security Limitations

The Logstash HTTP input plugin is a basic listener. It does not provide robust authentication or SSL/TLS termination by default. To secure this endpoint in production:

• Reverse Proxy: Place Nginx or HAProxy in front of Logstash to handle SSL termination and API key validation.
• Firewalling: Use iptables or cloud security groups to restrict incoming traffic to only the known IP addresses of the webhook provider.

Rollback and Recovery

Since this configuration only changes the Logstash pipeline state and does not modify system binaries, rollback is straightforward:

1. Stop the Logstash process (Ctrl+C or systemctl stop logstash).
2. Revert the .conf file to the previous version or remove the http input block.
3. Restart Logstash.