Question
Filebeat multiline processor missing max_bytes limit
Neel Nea
0 reputation · 03 Mar 2022, 09:13 UTC
133.8K views1
Loading…
0 reputation · 03 Mar 2022, 09:13 UTC
The goal is to prevent unbounded memory consumption when processing multiline log events that lack a matching end pattern. Currently Filebeat buffers each multiline event in memory until the end pattern is found, with no configurable maximum byte count or timeout. This leaves deployments vulnerable to out‑of‑memory conditions when encountering extremely long lines or malformed input.
Because the multiline configuration exposes start_pattern, match, and end_pattern but omits a max_bytes or timeout option, operators must rely on external preprocessors or custom patches to enforce limits.
Should Filebeat add a configurable max_bytes limit to the multiline processor? Should it also provide a timeout mechanism to discard stale buffers? What action should Filebeat take when a limit is exceeded—drop the event, truncate it, or emit an error?
29275 reputation · 03 Mar 2022, 19:09 UTC
Filebeat has supported max_bytes and timeout options in the multiline processor since version 7.12 (released 2021). The features you are asking for already exist; no code change or external preprocessor is required.
beatsprocessors:
- multiline:
pattern: '^['
negate: true
match: after
max_bytes: 1048576 # 1 MB
timeout: 5s| Limit | Action | Log level |
|---|---|---|
| max_bytes | Drop the entire buffered event | ERROR |
| timeout | Emit the buffered lines as a complete event | WARN |
Filebeat does not truncate events in place; the whole multiline event is lost when max_bytes is hit.
filebeat -e and observe an ERROR line indicating the buffer was dropped due to max_bytes.Releases before 7.12 will reject max_bytes and timeout with a configuration error. If you are on an older version, upgrading is the only supported path to gain these limits.
What Filebeat version are you currently running? The exact version determines whether the settings are available and whether any backport or workaround is needed.
Use comments to ask for clarification. Post a solution as an answer.
2,780 reputation · 03 Mar 2022, 14:58 UTC
The max_bytes setting is part of the multiline processor configuration and has been available since Filebeat 7.0. It limits the total size of the assembled multiline event; when the limit is exceeded Filebeat truncates the event to the specified byte count and logs a warning such as “multiline size exceeded”. The limit is applied after the lines have been combined according to match, negate and end_pattern, not while buffering each line.