Diagnosing Harvester Log Forwarding Failures to Loki or Elasticsearch

Recognizable condition

After deploying Harvester, you notice that no new log entries appear in your Loki or Elasticsearch cluster. The Harvester pod may also restart frequently, and its logs contain messages such as "connection refused", "timeout", or authentication failures.

Cause / diagnostic table

Symptom	Likely cause
No logs arrive, Harvester logs show "connection refused" or "timeout"	Network connectivity blocked (firewall, NetworkPolicy, wrong host/port)
Harvester logs contain "unauthorized" or "invalid credentials"	Incorrect authentication details (username/password, token, TLS cert)
Harvester exits with config‑parsing errors or silent failures	Malformed Filebeat/YAML section or using a schema from a different Harvester version
Logs appear intermittently or after a delay	TLS handshake issues (self‑signed cert, missing CA) or excessive log level causing noise

Ordered checks

1. Verify Harvester configuration syntax

   Run the validation command inside the Harvester pod or on the host where the config resides:

   harvester validate-config --config /etc/harvester/harvester.yaml

   Look for any validation errors. If the command is not available, check the Harvester version; the validate‑config subcommand was added in v0.5.

2. Inspect Harvester logs for specific error messages

   Use kubectl logs -l app=harvester --tail=100 (adjust label as needed) and search for:

   • "connection refused"
   • "timeout"
   • "unauthorized"
   • "tls: handshake failure"
   • YAML parsing errors

   Note the timestamp; it should correlate with the observed missing logs.

3. Test network reachability from the Harvester pod

   Execute a temporary shell in the pod:

   kubectl exec -it <harvester-pod> -- sh
   # inside the pod
   curl -v http://<loki-host>:<port>/ready
   # or for Elasticsearch
   curl -v https://<es-host>:<port>/_cluster/health
   

   Replace placeholders with the host and port defined in Harvester’s output section. A successful HTTP 200 (or TLS handshake) indicates basic connectivity. If you see "Connection refused" or timed out, the network path is blocked.

4. Validate authentication and TLS settings

   If the curl test succeeds but Harvester still fails, check the auth block:

   • Username/password: ensure they match the target system’s credentials.
   • API token: verify it is not expired and has sufficient scopes.
   • TLS: if using self‑signed certs, either add the CA to Harvester’s trusted store or set insecure_skip_verify: true (only for testing).

   You can test TLS directly with:

   openssl s_client -connect <es-host>:<port> -servername <es-host>

   Look for "Verify return code: 0 (ok)". Any other code indicates a trust issue.

5. Check for conflicting NetworkPolicy or firewall rules

   List NetworkPolicies in the namespace:

   kubectl get networkpolicy -n <namespace>

   Ensure there is an egress rule allowing traffic to the Loki/Elasticsearch service IP and port. If you use an external firewall, confirm that the node’s security group permits outbound TCP traffic on the configured port.

Fixes tied to findings

• Configuration syntax error – Correct the YAML according to the Harvester version’s schema (see release notes). Reapply the config and restart the Harvester daemon.
• Network block – Add an egress rule to the NetworkPolicy or adjust the cloud firewall to allow traffic from the Harvester pod’s CIDR to the target host/port. Verify with the curl test again.
• Authentication failure – Update the username/password, token, or TLS certificate in the Harvester config. If using Kubernetes secrets, ensure the secret is mounted correctly and the file permissions are readable by the Harvester process.
• TLS handshake issue – Either add the CA certificate to /etc/ssl/certs inside the container (or mount it) and point Harvester to it, or temporarily set insecure_skip_verify: true for troubleshooting only.
• Wrong host/port – Edit the output section to match the exact hostname (or service name) and port used by Loki or Elasticsearch. Remember that Kubernetes internal DNS uses <service>.<namespace>.svc.cluster.local.

Escalation criteria

If after completing all checks and applying the corresponding fixes:

• Logs still do not appear in Loki/Elasticsearch,
• Harvester continues to crash or restart, or
• You observe new error messages not covered in the table (e.g., "pipeline broken" or "queue full"),

then escalate to the platform team with the following information:

1. Harvester version (harvester version).
2. Relevant snippet of the harvester.yaml output section (redact secrets).
3. Recent Harvester logs (last 200 lines) showing the persistent error.
4. Results of the network connectivity test (curl -v output).
5. Any NetworkPolicy or firewall rules that affect the pod’s egress.

Providing this data will allow the team to investigate deeper issues such as backend overload, index mapping conflicts, or Harvester bugs.