Using the Filebeat Nginx Module to Ship Access Logs to Elasticsearch

Quick answer

Enable the Filebeat Nginx module, point it at your /var/log/nginx/access.log* files, and restart Filebeat. The module ships the logs to Elasticsearch using the built‑in nginx-access-log ingest pipeline, which extracts fields such as client IP, timestamp, request method, URL, status code, and response size into the index pattern filebeat-*-nginx.access.

How the module works

The Nginx module is a Filebeat module that bundles:

• A fileset defining the log paths and the ingest pipeline to use.
• The Elastic Ingest Node pipeline nginx-access-log, which contains Grok patterns for the combined log format.

When Filebeat starts, it reads the module configuration, registers the pipeline with the connected Elasticsearch cluster, and then forwards each log line to that pipeline for parsing before indexing.

Worked configuration

# filebeat.yml
filebeat.modules:
  - module: nginx
    access:
      enabled: true
      var.paths: ["/var/log/nginx/access.log*"]
output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]
  index: "filebeat-%{[agent.version]}-nginx.access"

Place the snippet in /etc/filebeat/filebeat.yml (or the location used by your installation) and restart the service:

# Run as root or with sudo
sudo systemctl restart filebeat

Verification steps

1. Check that Filebeat can reach Elasticsearch:

   filebeat test output

   Look for a line like OK or a successful connection message.
2. Generate a test log line (you can do this on the Nginx host):

   echo '127.0.0.1 - - [10/Oct/2023:13:55:36 +0000] "GET /index.html HTTP/1.1" 200 1024' | sudo tee -a /var/log/nginx/access.log

3. After a few seconds, query Elasticsearch for the newest document:

   GET filebeat-*-nginx.access/_search?size=1&sort=@timestamp:desc

   The response should contain fields such as client.ip, http.request.method, url.path, http.response.status_code, and http.response.body.bytes.
4. If the fields are missing, check the Filebeat logs for errors related to pipeline loading or file access.

Limits and common pitfalls

Log format expectations

The built‑in pipeline assumes the combined log format:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined

If you use a custom log_format in nginx.conf, the pipeline will fail to extract fields, resulting in raw message fields only. To fix this you have two options:

• Override the pipeline by supplying a custom ingest pipeline that matches your format, and set var.pipeline in the module config.
• Revert Nginx to the combined format or add a second log file that uses the combined format for Filebeat to consume.

Version requirements

The Nginx module was introduced in Filebeat 7.0. Earlier releases (6.x and below) do not contain the module; you would need to rely on Logstash with a Grok filter or manually define an ingest pipeline.

File permissions

Filebeat runs under the filebeat user (or root if you started it that way). If that user cannot read the Nginx log files, Filebeat will silently skip them and no data will be shipped. Verify permissions:

sudo -u filebeat test -r /var/log/nginx/access.log && echo "readable" || echo "not readable"

Adjust ownership or add the filebeat user to a group that has read access, e.g., sudo usermod -a -G adm filebeat on systems where Nginx logs are group‑readable by adm.

Performance considerations

Parsing occurs on the Elasticsearch ingest node, so the Filebeat agent itself adds little CPU overhead. However, very high request rates can increase indexing latency and storage pressure. Mitigation strategies include:

• Enabling Index Lifecycle Management (ILM) to roll over indices based on size or age.
• Increasing the refresh interval on the filebeat-*-nginx.access index during bulk ingest periods.
• Monitoring the ingest node pipeline latency via the Elasticsearch /_nodes/stats/ingest API.

Summary

The Filebeat Nginx module gives you a zero‑code way to ship and parse Nginx access logs into Elasticsearch. Enable the module, point it at your log files, restart Filebeat, and verify that fields like client.ip and http.response.status_codefilebeat-*-nginx.access index. Remember that the module expects the combined log format, requires Filebeat 7.0+, and needs the Filebeat user to have read access to the log files. Adjust the pipeline or log format if you deviate from these assumptions, and keep an eye on ingest node load when log volumes grow.