Can kubectl logs --previous reliably capture output from a CrashLoopBackOff container before rotation?
0 reputation · 22 Mar 2024, 10:34 UTC
0 reputation · 22 Mar 2024, 10:34 UTC
I am troubleshooting a Deployment whose Pods enter CrashLoopBackOff shortly after starting. I know the kubelet writes container stdout and stderr to files under /var/log/pods on the node, and that kubectl logs proxies those files through the API server, with --previous exposing the last terminated container instance.
What I cannot pin down is the reliability boundary. The kubelet rotates logs based on containerLogMaxSize and containerLogMaxFiles settings, and if the container restarts rapidly, the previous instance's file may be rotated away or overwritten before I can fetch it. I am also unsure whether rotation behavior differs when the failing container produces very large output, since proxying big files through the API server is known to be slow.
Assume a recent Kubernetes version (1.28+) with default kubelet log rotation settings and no external log aggregator.
Does --previous always survive at least one restart cycle, or can rotation delete it before retrieval? At what log volume or restart frequency does this become unreliable? When should I switch to node-level file access or a logging agent instead?
32275 reputation · 22 Mar 2024, 13:37 UTC
No, --previous is not guaranteed. It reads the log file of the most recently terminated container instance, and that file survives only until kubelet rotation deletes it. In practice, with default rotation settings and typical crash output, it reliably survives at least one restart cycle — often many — but a container that crashes fast and logs heavily can rotate its own previous-instance file away within a few restarts, before you fetch it.
Confirmed mechanics: the kubelet writes each container instance's stdout/stderr to a file under /var/log/pods/<pod_uid>/<container>/, symlinks it from /var/log/containers, and rotates per container log file. kubectl logs --previous maps to the last terminated instance's file via the API server. Rotation is governed by the kubelet's containerLogMaxSize (default 10Mi) and containerLogMaxFiles (default 5).
The likely (not guaranteed) part is the arithmetic: with defaults, roughly 50Mi of total output across restarts can push the oldest file past retention. Exact per-file accounting has varied across releases, so treat byte counts as approximations and verify against your nodes.
--previous stays fetchable for a long time.--previous returns nothing regardless of rotation settings.kubectl logs --previous --tail=200 or --limit-bytes=1Mi.kubectl logs <pod> -c <container> --previous --tail=500. Repeat right after each restart if the loop is fast./var/lib/kubelet/config.yaml or the kubelet configz endpoint for containerLogMaxSize/containerLogMaxFiles.--previous comes up empty or truncated, read the rotated files directly on the node: ls -l /var/log/pods/<pod_uid>/<container>/ shows the numbered files; the highest number is the current instance.Low-output crashers: --previous is effectively reliable; use it. Verbose crashers in a tight loop: treat --previous as best-effort, fetch fast with --tail, and fall back to node-level file access. Anything you need for audit or recurring diagnosis: use a logging agent, because node-local files are ephemeral by design.
One caveat worth flagging: if your cluster sets non-default rotation values, the volume thresholds shift proportionally, so the config check in step 2 is what turns this from a heuristic into a reliable boundary.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.