[Logstash::Outputs::ElasticSearch] Failed to authenticate user … – credential reload without pipeline restart

The Logstash Elasticsearch output plugin resolves credentials at pipeline startup and does not natively monitor the keystore for changes during a running session. When username, password, or API key values are updated via the keystore, the plugin continues using the cached reference, producing authentication errors such as '[Logstash::Outputs::ElasticSearch] Failed to authenticate user ...' until the pipeline is reloaded or Logstash is restarted.

The plugin caches keystore values after the initial pipeline parse; no runtime hook watches for encrypted secret updates.

Logstash provides SIGHUP and a _reload API endpoint, both of which trigger a full pipeline restart, risking in-flight event loss. The plugin’s retry logic handles transient connection failures but does not re‑resolve keystore‑bound credentials after secret rotation.

1. Check your Logstash version with logstash --version and the Elasticsearch output plugin version via logstash-plugin list | grep elasticsearch; behavior may differ across 7.x, 8.x, and later releases.
2. Add or update a test entry in the keystore using logstash-keystore add and send SIGHUP or call the _reload API; observe whether authentication succeeds. If not, a full Logstash restart is required.
3. For automated least‑privilege rotation without downtime, externalize credentials as environment variables or mounted secrets (e.g., Kubernetes secrets, HashiCorp Vault) and reference them in the output configuration, noting that the process still caches its environment at launch.
4. If a pipeline reload is necessary, perform it during a quiet period and measure event processing gaps or loss to assess impact on in-flight events.

Missing diagnostic detail: Which Logstash version and Elasticsearch output plugin version are you using? This determines whether a SIGHUP re-reads the keystore or if a full restart is required.