Diagnosing Beats TLS/SSL Certificate Validation Errors When Shipping to Elasticsearch

Recognizable condition

The Beats process (Filebeat, Metricbeat, etc.) exits or repeatedly logs errors such as:

• failed to connect to Elasticsearch: x509: certificate signed by unknown authority
• TLS handshake failure or connection refused after enabling SSL/TLS.
• No new documents appear in Elasticsearch indices while the Beats module continues to read files.
• Internal metrics show rising retrying connection counts and occasional CPU spikes.

Cause / diagnostic table

Symptom	Likely cause
x509: certificate signed by unknown authority	Beats cannot verify the Elasticsearch server certificate against the configured CA bundle.
TLS handshake failure after config change	Incorrect CA path, missing intermediate certificates, or a self‑signed cert not trusted.
Beats reads files but no data in Elasticsearch	Output plugin is failing silently; Beats keeps retrying.
High retry metric and CPU usage	Repeated connection attempts due to TLS validation errors.

Ordered checks

1. Verify Beats can read the CA bundle

   Run the config test as the Beats user (or with sudo if the Beats service runs under a dedicated account):

   sudo -u beats ./filebeat test config -c /etc/filebeat/filebeat.yml

   Look for the line:

   SSL section: certificate verification: OK

   If the test fails or the line is missing, the CA bundle path is likely wrong or unreadable.

2. Check the CA bundle contents

   Ensure the file is absolute, readable, and contains the root CA that signed the Elasticsearch certificate:

   ls -l 
   openssl x509 -in  -text -noout | head -20

   Replace with the path from output.elasticsearch.ssl.certificate_authorities.

3. Validate the Elasticsearch server certificate chain

   From the Beats host, test the TLS handshake directly:

   openssl s_client -connect :9200 -showcerts -CAfile 

   Look for:

   • Verify return code: 0 (ok)
   • Certificate chain ending with a trusted root.

   Any other verify code indicates a mismatch between the server cert and the CA bundle.

4. Confirm Beats output plugin settings

   Inspect the relevant section in filebeat.yml (or the respective Beats config):

   output.elasticsearch:
     hosts: [":9200"]
     ssl.certificate_authorities: [""]
     # ssl.verification_mode: full   # default in 8.x; can be set to "strict"

   Ensure hosts matches the hostname used in the openssl s_client test.

5. Restart Beats and monitor

   After correcting the CA bundle or adjusting verification mode, restart the service:

   sudo systemctl restart filebeat

   Watch the logs for a few seconds:

   sudo journalctl -u filebeat -f

   Expected log lines:

   • Published X events
   • output.elasticsearch.success metric incrementing
   • Absence of x509: certificate signed by unknown authority or handshake errors.

Fixes tied to findings

• Incorrect or unreadable CA bundle – Correct the file path to an absolute location, ensure the Beats user has read permission (chmod 640 and appropriate group ownership).
• Missing intermediate certificates – Concatenate the root and any intermediate CA certificates into the bundle file, ordered from leaf to root.
• Self‑signed or internal CA not trusted – Add the self‑signed certificate to the CA bundle, or (for testing only) set ssl.verification_mode: none. Remember to revert this in production.
• Hostname mismatch – If using IP addresses in hosts while the certificate contains a DNS name, either change the host setting to match the certificate’s Subject Alternative Name or regenerate the certificate with the appropriate IP SAN.
• Beats version change (7.x → 8.x) – The default verification mode became strict. If you previously relied on the more permissive full mode, explicitly set ssl.verification_mode: full to retain the old behavior, or update your CA bundle to meet strict validation.

Escalation criteria

• After verifying the CA bundle and server certificate, Beats still logs TLS handshake failures.
• The openssl s_client test returns a verify code other than 0, indicating a chain issue that cannot be resolved by adjusting the bundle.
• Beats continues to show rising retry metrics despite correct configuration, suggesting a network‑level block (firewall, SELinux, or Elasticsearch refusing connections).
• In these cases, collect:
• Beats log snippet (journalctl -u filebeat)
• Output of ./filebeat test config
• Output of openssl s_client -connect :9200 -showcerts
• Elasticsearch HTTP logs (/var/log/elasticsearch/*.log) for connection attempts.
• Escalate to the Elasticsearch administration team or network/security team with the above data for further investigation.