Harvester 1.6.0: HTTP/2 Transition for Elastic Output – Unresolved Retry Backoff Behavior

Retry Backoff Parameters

In Harvester 1.6.0, the elastic output plugin uses hard-coded values for its exponential backoff algorithm. These parameters are not exposed via the public configuration or environment variables. Based on the RetryManager implementation, the default behavior is as follows:

• Initial Delay: 1 second
• Multiplier: 2
• Jitter: ±25% of the computed delay

The formula applied is: (initialDelay * multiplier^attempt) + jitter.

Configuration and Overrides

There is currently no mechanism to override the multiplier or jitter values through the configuration file or environment variables. Attempting to define properties such as backoff.multiplier or backoff.jitter will have no effect, as the plugin ignores these keys in favor of the hard-coded constants.

HTTP/1.1 vs. HTTP/2 Behavior

While the mathematical timing of the backoff is identical across both protocols, the impact on the transport layer differs significantly:

Metric	HTTP/1.1	HTTP/2
Retry Timing	Exponential (1s, 2s, 4s...)	Exponential (1s, 2s, 4s...)
Resource Usage	New TCP connection per retry	New bidirectional stream per retry
Exhaustion Risk	Socket exhaustion (Ephemeral ports)	Stream limit exhaustion (MAX_CONCURRENT_STREAMS)

Under HTTP/2, connection pool exhaustion may occur more rapidly if the Elasticsearch cluster has a low limit on concurrent streams, as multiple retries may attempt to open streams on the same connection before the backoff interval expires.

Verification Steps

To verify the actual backoff intervals in your specific environment, use the following method:

1. Set the logging level for the elastic output plugin to DEBUG or TRACE.
2. Trigger a connection failure (e.g., via a firewall rule or by pausing the Elasticsearch service).
3. Monitor the logs for RETRY_BACKOFF messages to observe the timestamp intervals between attempts.

Note: To provide a more specific recommendation on stream exhaustion, please provide the max_concurrent_streams setting of your Elasticsearch HTTP/2 implementation.