Teleport Log Sink regex filtering and high-volume event latency

Question

Teleport utilizes Log Sinks to stream audit events to external providers like Webhooks, Slack, or PagerDuty. To manage notification noise, administrators apply regex filters within the sink configuration to exclude high-frequency events, such as routine health heartbeats or repetitive session metadata.

p>There is uncertainty regarding the performance impact of these filters when the Audit Service processes high-volume log streams. While filtering reduces the volume of downstream notifications, it is unclear if complex regex processing at the sink level introduces measurable latency for critical security events during an incident spike.
How does the Teleport Audit Service prioritize event delivery when multiple regex filters are applied to high-throughput streams? Is there a documented mechanism to implement rate-limiting at the sink level to prevent webhook API exhaustion during event storms?

Tasadduq Burney · Answer

Event Prioritization and Regex Processing
The Teleport Audit Service does not implement a priority queue for event delivery based on the content of the event or the complexity of the regex filter. Events are processed sequentially as they are generated. When multiple regex filters are applied to a high-throughput stream, the processing time for each event increases linearly with the number of filters and the complexity of the expressions.

Performance Impact of Regex Filtering
While regex filtering reduces the volume of data sent to external providers, the evaluation occurs within the event pipeline. If complex patterns—specifically those prone to catastrophic backtracking—are used, CPU utilization on the Teleport Auth server will spike. This can introduce measurable latency for all events, including critical security alerts, because the processing bottleneck occurs before the event is dispatched to the sink.

Rate Limiting and Webhook Exhaustion
Teleport does not currently provide a built-in, configurable rate-limiting mechanism at the individual Log Sink level to prevent external API exhaustion (e.g., Slack or PagerDuty rate limits). To prevent webhook exhaustion during an event storm, you must implement one of the following architectural patterns:

Intermediate Proxy/Buffer: Route Teleport Log Sinks to an intermediate collector (such as Fluentd, Logstash, or a custom lightweight proxy) that supports token-bucket or leaky-bucket rate limiting.
  Aggressive Filtering: Refine regex patterns to be as specific as possible, avoiding greedy operators (.*) that force the engine to scan large payloads.

Verification and Monitoring
To determine if regex processing is impacting your event latency, monitor the following metrics during a simulated event spike:

CPU Utilization: Check for spikes on the Teleport Auth server coinciding with high event volume.
  Delivery Lag: Compare the event_timestamp in the audit log with the arrival timestamp at the webhook destination.

Diagnostic Requirement: To provide a more specific recommendation on filter optimization, please provide the specific regex patterns currently in use and the average event payload size.

Teleport Log Sink regex filtering and high-volume event latency

1 answer

Event Prioritization and Regex Processing

Performance Impact of Regex Filtering

Rate Limiting and Webhook Exhaustion

Verification and Monitoring

Your perspective belongs here.

1 question comment

Per‑Sink Worker Regex Execution