Resolving 'No space left on device' Errors in Docker Environments

The Problem: Disk Exhaustion in Docker

When a Docker host runs out of disk space, you will typically encounter the No space left on device error during a docker build or when a container attempts to write to its writable layer. This often happens even when the host OS reports available space, because the Docker root directory (usually /var/lib/docker) resides on a partition that has reached its capacity due to accumulated build artifacts and orphaned data.

Diagnostic Matrix: Identifying the Storage Leak

Before running destructive cleanup commands, identify which Docker object type is consuming the most space.

Symptom	Likely Cause	Diagnostic Command
Builds fail at late stages	Dangling image layers	docker images -f "dangling=true"
Slow disk I/O, high inode usage	Accumulated stopped containers	docker ps -a
Host disk full, but images are few	Orphaned anonymous volumes	docker volume ls
Rapid growth after CI/CD runs	BuildKit cache accumulation	docker system df

Step-by-Step Recovery Process

Follow these checks in order to reclaim space without accidentally deleting critical persistent data.

1. Analyze Current Usage

Run the following command on the Docker host to see a breakdown of space used by images, containers, and volumes:

docker system df

Check the RECLAIMABLE column. If this percentage is high, the following cleanup steps will be effective.

2. Clear Dangling Images

Dangling images are layers that no longer have a relationship to any tagged image, often created when you rebuild an image with the same tag. These are safe to remove.

# Run as user with docker group permissions or root
docker image prune

Risk: Low. Only untagged images are removed.

3. Remove Stopped Containers

Containers that have exited still occupy space in the writable layer and may hold onto allocated logs.

docker container prune

Risk: Low, provided you do not need to inspect the logs of stopped containers for debugging.

4. Purge Unused Volumes

Anonymous volumes (volumes without a specific name) often persist after a container is deleted. These are the most common cause of "stealth" storage growth.

docker volume prune

Risk: High. This deletes all volumes not currently used by at least one container. Ensure your database data is stored in named volumes or bind mounts before running this.

The "Nuclear" Option: Full System Prune

If individual prunes do not reclaim enough space, you can perform a comprehensive cleanup. This is common in CI/CD runners where the environment is ephemeral.

# Removes all stopped containers, unused networks, and all images without at least one container associated with them
docker system prune -a --volumes

Caution: This will force Docker to re-download every image the next time you start your services, which may increase deployment time and network bandwidth usage.

Verification and Validation

To verify the fix, compare the host partition state and the Docker internal state:

1. Run df -h /var/lib/docker (or the path configured in docker info) to verify the host OS sees the freed blocks.
2. Run docker system df to ensure the reclaimable percentage has dropped.
3. Attempt the operation that previously failed (e.g., docker build) to confirm the error is resolved.

Escalation Criteria

If the No space left on device error persists after a system prune -a --volumes, the issue is likely not Docker-internal. Escalate to system administration for the following:

• Inode Exhaustion: Run df -i. If the inode count is 100% but disk space is available, you have too many small files (common with some node_modules setups).
• Partition Resizing: The /var/lib/docker partition is physically too small for the workload requirements.
• Log Rotation: Check /var/lib/docker/containers/<id>/<id>-json.log. If a single log file is several gigabytes, implement log-opts in daemon.json to limit log size.