Diagnose Beats Connection Refused and Publishing Errors to Elasticsearch

Beats logs show repeated Connection refused or publishing timeouts

When Filebeat or Metricbeat cannot ship events, the agent log repeatedly shows publishing failures such as "Failed to publish events" with "connection refused", "dial tcp: connect: connection refused", or "context deadline exceeded". The useful takeaway is that the error is almost always network reachability, output configuration mismatch, or a rejecting Elasticsearch cluster, not a bug in the Beat itself. Fix the layer where the connection is blocked first, then verify the Beat recovers without changing bulk settings.

Condition and likely causes

Symptom in Beat logs	Most likely layer	Typical cause
connection refused	Network / Output config	Wrong host/port in output.elasticsearch, service not listening, firewall/security group blocks outbound from Beat host
timeout / context deadline exceeded	Network / Cluster load	Network latency or packet loss, Elasticsearch overloaded or unresponsive
503 Service Unavailable or circuit_breaking_exception	Cluster health	Elasticsearch in red/yellow, disk watermark breached, circuit breaker tripped
x509 certificate error	TLS config	Protocol mismatch HTTP vs HTTPS, invalid CA, verification_mode set incorrectly

Ordered checks

1. Confirm the current error from the Beat host

Run the Beat in foreground to see live output. Run as the service user that normally runs the Beat.

/usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml

Expected check: look for the output.elasticsearch section in the error and the target host:port. Do not assume the error is persistent until you see it repeat across multiple publish attempts.

Risk: -e logs to stdout and can be verbose. Run in a terminal with limited time.

2. Verify network reachability to the configured endpoint

From the Beat host, test TCP connectivity to the host and port defined in output.elasticsearch.

nc -vz <ES_HOST> <ES_PORT>

Alternative with telnet:

telnet <ES_HOST> <ES_PORT>

Required permissions: outbound network access from the host. Expected check: connection succeeds. If it fails, the problem is network routing, security group, or the Elasticsearch node is down.

3. Validate beats.yml output configuration

Open the Beat config and check the output block. Common mismatches are protocol, port, and hostname.

output.elasticsearch:
  hosts: ["https://<ES_HOST>:9200"]
  username: "<USER>"
  password: "<PASSWORD>"
  ssl.verification_mode: full

Check for syntax errors such as tabs vs spaces, duplicate output sections, or using http when the cluster requires https. Unfamiliar term: verification_mode controls whether the Beat validates the Elasticsearch TLS certificate against a CA.

Risk: disabling verification_mode to none bypasses certificate validation and should be avoided in production.

4. Check Elasticsearch cluster health from the Beat host

Use curl to query the cluster API from the same host that runs the Beat.

curl -u <USER>:<PASSWORD> https://<ES_HOST>:9200/_cluster/health

Expected check: status is green or yellow and the endpoint responds quickly. If the request hangs or returns 503, the cluster is rejecting writes.

Inspect Elasticsearch logs for circuit_breaking_exception or disk watermark messages. These indicate the cluster is under memory or disk pressure and will refuse indexing requests.

Fixes tied to findings

Network blocked or wrong host/port

If nc/telnet fails but the config looks correct, update firewall rules or security groups to allow outbound TCP from the Beat host to the Elasticsearch port, default 9200. Then update beats.yml with the correct host address and port, ensuring the protocol matches the cluster's SSL configuration.

Configuration mismatch

If the config uses http for an https cluster or the port is wrong, correct the hosts list and protocol. Restart the Beat service after the change.

systemctl restart filebeat

Cluster rejecting writes

If curl shows red status or 503, do not increase bulk_max_size. Increasing bulk_max_size during a connection crisis can exacerbate memory pressure on a struggling cluster. Instead, free disk space to get below the flood stage watermark, or address the circuit breaker cause with infrastructure support.

Verification

After a change, run the Beat with -e again and confirm the log transitions from publishing errors to normal harvester and publish messages. Also re-run the curl health check from the Beat host to confirm the cluster is reachable and responsive.

Escalation criteria

Escalate to Infrastructure/DevOps if:

• Network connectivity is confirmed with nc/telnet and curl succeeds, but Beats still receive 503 Service Unavailable consistently.
• Elasticsearch logs show repeated circuit_breaking_exception or disk watermark breaches.
• Certificate errors persist after correcting hosts and verification_mode.

Provide the Beat host name, the exact hosts entry from beats.yml, the output of nc and curl, and relevant Beat log lines.

Limitations

This guide assumes a standard output.elasticsearch configuration in Beats 7.x/8.x. Custom outputs, proxy layers, or load balancers change the diagnostic path. Verification steps depend on current cluster settings and network policies, which require manual confirmation in your environment.