Filebeat autodiscover: multi‑level glob pattern support in nested modules

Resolving Multi-level Path Matching in Filebeat

To match files across multiple directory levels in Filebeat autodiscover, you must use the recursive wildcard **. A single asterisk * matches only within a single path segment and will not traverse subdirectories.

Correcting Glob Patterns

If your current configuration uses /var/log/*/logs/*.log, it will only find files exactly two levels deep. To capture all .log files regardless of depth within a specific root, use the following syntax in your autodiscover template:

paths:
  - "/var/log/app/**/*.log"

Ensuring exclude_files is Respected

The exclude_files option is a per-input setting. In an autodiscover configuration, it must be placed inside the generated input definition within the template, not at the provider or top-level configuration. If it is placed outside the config block of the template, Filebeat ignores it.

Additionally, exclude_files follows the same glob rules as paths. To exclude files in nested directories, you must also use the recursive wildcard:

config:
  - type: log
    enabled: true
    paths:
      - "/var/log/app/**/*.log"
    exclude_files: ['\.gz$', '.*\.tmp', '**/test/*.log']

Provider Evaluation and Determinism

Filebeat autodiscover providers (such as Docker, Kubernetes, or local hints) operate independently and concurrently. There is no documented global order of evaluation between different providers.

To guarantee deterministic ingestion and avoid duplicate logs, you must ensure that your conditions are mutually exclusive. If a single event matches multiple templates across different providers, Filebeat may start duplicate inputs for the same file. Use specific labels or namespaces in your conditions to isolate which provider handles which path.

Verification and Debugging

To verify which files are being matched and which templates are expanding, run Filebeat with debug logging enabled for the autodiscover and input components:

filebeat -e -d "autodiscover,input,harvester"

Note: Recursive scans of very large directory trees can increase startup time and resource usage. It is recommended to pair ** patterns with ignore_older to limit the number of files the harvester tracks.

To provide a more specific template recommendation, please clarify: Are you using a hints-based provider or a template-based provider (e.g., Kubernetes/Docker)?