Diagnosing Jupyter Notebook Kernel Crashes from Memory Exhaustion

Recognizable Condition

When a Jupyter Notebook kernel repeatedly dies, you may see a 'Killed' message in the terminal where the notebook server was started, or a popup in the notebook UI indicating the kernel has died. The notebook output stops updating and any further cell execution fails.

Cause and Diagnostic Overview

Condition	Likely Cause	What to Look For
Kernel dies with 'Killed'	Process exceeds available RAM or cgroup memory limit → OOM kill	OOM messages in dmesg or kern.log; memory usage spiking before death

Ordered Checks

1. Identify the kernel process: Run ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | grep -i jupyter (no sudo needed for user‑owned processes). Note the PID and resident memory (%MEM).
2. Monitor memory while reproducing the workload: Use top -p <PID> or htop -p <PID> and watch the RES column climb as you run the problematic cell.
3. Check for OOM kill messages: After a crash, run dmesg | grep -i 'killed process' or sudo grep -i 'out of memory' /var/log/kern.log. Look for a line that mentions the Jupyter kernel’s command or PID.
4. Verify cgroup limits (if running in a container or systemd slice): cat /sys/fs/cgroup/memory/<slice>/memory.limit_in_bytes. Compare the limit to the observed usage.

Fixes Tied to Findings

Finding	Recommended Fix	How to Apply
Memory usage approaches or exceeds the available RAM/cgroup limit	Reduce the notebook’s memory footprint	Downsample large arrays, use chunked readers (e.g., pandas.read_csv(..., chunksize=100000)), or delete intermediate variables with del and gc.collect().
Kernel runs in a container with low memory limit	Increase the container’s memory limit	For Docker: docker run -m 4g ...; for Kubernetes: set resources.limits.memory: 4Gi in the pod spec. Test in a staging namespace first.
System has spare RAM but the kernel is constrained by a low user limit	Adjust the user’s memory limit or enable swap	Edit /etc/security/limits.conf to raise memlock or as for the user, or create a swap file: sudo fallocate -l 2G /swapfile && sudo chmod 600 /swapfile && sudo mkswap /swapfile && sudo swapon /swapfile. Monitor swap usage with free -h.

Escalation Criteria

• The notebook consistently exceeds 80 % of total host RAM even after optimizations.
• Increasing memory limits causes system‑wide swapping that degrades other services.
• OOM kill messages persist despite raising limits, indicating a possible leak or unbounded data growth.
• You lack permission to adjust cgroup or system limits on the host.

In these cases, contact the system administrator or consider moving the workload to a cluster with larger memory nodes, or redesign the analysis to use out‑of‑core libraries (e.g., Dask, Vaex).