Answer
Kubernetes does not retain container logs after a pod is deleted; logs are stored only on the node’s filesystem and are removed by the node’s log‑rotation policy. There is no built‑in API to access logs for terminated pods.
Likely explanation
When a pod exits due to CrashLoopBackOff, the container runtime discards its stdout/stderr buffers. Deleting the pod removes the container from the runtime, so kubectl logs returns empty unless the logs were forwarded elsewhere before termination.
Confirmed facts
- Kubernetes does not store container logs by default; logs are ephemeral and tied to the container lifecycle on the node.
- A node‑level logging agent (e.g., Fluentd, Fluent Bit, Filebeat) running as a DaemonSet can capture logs from the container’s log files and ship them to external storage before the container is removed.
- Without such an agent, logs appear to vanish after pod deletion because nothing retains them durably.
How to reliably retain logs without external tooling
Pure Kubernetes does not provide a native, API‑accessible log persistence layer. To keep logs you must either:
- Deploy a logging DaemonSet that reads
/var/log/pods/* (or /var/log/containers/*) and forwards logs to a durable store (Elasticsearch, Loki, etc.).
- Add a sidecar container to each workload that writes stdout/stderr to an
emptyDir volume backed by a PersistentVolumeClaim; the PVC survives pod deletion, preserving the logs.
Both approaches require extra components; there is no way to retain logs using only the Kubernetes API.
Trade‑offs: node‑level vs cluster‑wide log aggregation
| Aspect | Node‑level agent (DaemonSet) | Cluster‑wide solution (e.g., sidecar per pod) |
|---|
| Deployment overhead | One DaemonSet per node; low pod count | One sidecar per application pod; higher resource usage |
| Log collection reliability | Depends on node health; if node fails, logs on that node may be lost unless replicated | Logs follow the pod; if pod is evicted, sidecar can still ship logs before termination |
| Configuration complexity | Centralized agent config; easier to manage rotation/retention policies | Per‑pod config needed; more variables to track |
| Storage pressure | Agent can buffer and rotate logs locally before shipping | Sidecar writes to PVC; may consume more PV storage if not rotated |
Missing diagnostic detail
To give a precise recommendation, please confirm whether you already have a logging DaemonSet (e.g., fluent-bit, fluentd, filebeat) running in your cluster. If not, deploying one is the simplest way to retain CrashLoopBackOff logs.
kubernetes