Grafana Loki alert rule max_lines truncation behavior unspecified
0 reputation · 20 Feb 2024, 21:02 UTC
0 reputation · 20 Feb 2024, 21:02 UTC
When configuring a Loki-based alert rule in Grafana, the max_lines parameter limits the number of log lines returned for evaluation. The goal is to determine which subset of lines is kept when the actual result set exceeds this limit, as this decides whether critical error messages are included in the alert condition.
The truncation strategy—whether Grafana retains the earliest lines, the latest lines, or some other selection—is not documented, creating uncertainty about alert reliability and consistency across environments.
max_lines?30775 reputation · 21 Feb 2024, 04:58 UTC
When Grafana evaluates a Loki alert rule and the query returns more log lines than the configured max_lines value, the retained subset consists of the most recent entries up to that limit. Grafana forwards max_lines as the limit parameter to Loki’s query_range API. By default, Loki sorts results by timestamp descending (newest first) and returns the top N lines matching that order. Consequently, older lines beyond the first N are dropped.
This behavior applies consistently across the Loki query types supported in alert rules (raw log queries, metric queries, and joined queries) because Grafana applies the limit after Loki’s internal sorting, regardless of the evaluation interval. The alert window duration does not alter the order or limit; it only re‑runs the same query.
Users can influence which lines are kept by inserting an explicit sort or limit clause directly in the Loki query builder within the alert rule. Adding | sort asc reverses the default order, causing the oldest lines to be retained instead. Adding a separate | limit N clause before Grafana’s max_lines takes effect first; the effective limit becomes the smaller of the user‑specified value and max_lines.
| sort desc (or remove any existing sort) to confirm the default newest‑first retention.| sort asc and verify that the retained lines shift to the oldest entries in the window.| limit 50 to test how the combined limit interacts with max_lines.query_range request to see the exact limit parameter and sort order sent to Loki.Assumptions and uncertainty: The description above relies on Loki’s default sort order descending by timestamp. If a Loki deployment customizes the default sort direction, the truncation could reverse, though this is uncommon in standard installations. When other Grafana transformations (e.g., reducing functions, formatting) are applied after the query, the underlying line limit remains, but the visible subset in the alert annotation may differ. These edge cases are marked for review.
To fully confirm the behavior in your environment, run a test query with a known set of timestamps, set max_lines to a small value, and compare the returned lines against the expected newest entries. If the result set does not match the expected retention pattern, the missing diagnostic detail is the Loki server’s query‑handler configuration or any custom query transformations applied in the alert rule. Please provide that detail if it changes the recommended troubleshooting steps.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.