You Deleted the Docker Containers, but the Disk Is Still Full: Why, and How to Fix It
It was a Friday evening. Every pipeline on our shared CI runner turned red at the same time, all with the same message:
no space left on device
The team had already done the "obvious" thing. "We deleted all the old containers," someone said in the channel. I logged in and ran df -h. The disk was still 98% full.
This is one of the most common Docker problems I see on build servers, and in 14+ years of working with infrastructure it still catches good engineers. It is also a question I like to ask in DevOps interviews, because the answer shows whether someone understands how Docker actually stores things.
Why deleting containers doesn't free much space
A container is a thin, writable layer on top of an image. When you delete a container, you delete that thin layer, and usually not much else.
The space on a Docker host is mostly used by four other things:
- Images. Every build creates a new image. Old tags and untagged ("dangling") images stay on the disk until you remove them. Deleting a container does not delete the image it came from.
- Build cache. BuildKit keeps intermediate layers so the next build is fast. On a busy CI runner that builds many projects, this cache can grow to tens of gigabytes.
- Volumes. Named and anonymous volumes survive after the container is gone. Databases and test containers often leave them behind.
- Container logs. With the default
json-filelogging driver and no size limit, one noisy container can write logs until the disk is full.
So "I deleted the containers" is like taking the cups off the table and leaving the kitchen full of dirty dishes.
Step 1: Find out what is using the space
Don't start deleting blindly. First, ask Docker:
docker system df
You'll see something like this:
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 187 6 61.2GB 58.9GB (96%)
Containers 6 4 310MB 120MB (38%)
Local Volumes 42 3 9.8GB 9.1GB (92%)
Build Cache 1240 0 38.4GB 38.4GB
(These numbers are an illustration of what a busy runner typically looks like.) The RECLAIMABLE column tells you where the easy wins are.
For more detail:
docker system df -v # per image, volume and cache entry
du -sh /var/lib/docker/* # what is on disk, by Docker's own folders
If docker system df looks small but the disk is still full, check the logs:
sudo du -sh /var/lib/docker/containers/*/*-json.log | sort -h | tail
Step 2: Clean up on purpose
Each cleanup command has a clear scope. Use the one that matches what you found.
# Remove dangling (untagged) images
docker image prune
# Remove all images not used by a container (bigger cleanup)
docker image prune -a
# Remove build cache older than 3 days
docker builder prune --filter until=72h
# Remove volumes not used by any container
docker volume prune
# Everything unused at once - build servers only
docker system prune -a --volumes
A word of warning: docker system prune -a --volumes is fine on a CI runner where everything can be rebuilt. On a server that runs real workloads, it can delete a volume that holds data you care about. Always read the confirmation prompt.
For a huge log file, don't just rm it while the container is running; the process keeps the file open and the space is not freed. Truncate it instead:
sudo truncate -s 0 /var/lib/docker/containers/<id>/<id>-json.log
Step 3: Stop it from happening again
Cleaning up once is not a fix. These are the changes I make on every build host.
1. Limit log size in /etc/docker/daemon.json:
{
"log-driver": "json-file",
"log-opts": {
"max-size": "50m",
"max-file": "3"
}
}
Restart Docker after the change. New containers pick it up; old ones need to be recreated.
2. Cap the build cache. With BuildKit you can set a garbage-collection limit so the cache cleans itself:
{
"builder": {
"gc": {
"enabled": true,
"defaultKeepStorage": "20GB"
}
}
}
3. Schedule a prune job on CI runners, for example a nightly cron or systemd timer:
docker image prune -a -f --filter until=168h
docker builder prune -f --filter until=72h
4. Alert early. Set a disk alert at 80%, not 95%. At 95% you are already firefighting on a Friday evening.
5. Use ephemeral runners where you can. Auto-scaling or container-based runners that are thrown away after each job never collect old images in the first place.
The interview angle
When I ask "You removed the containers, but the disk is still full. Why?", here's what a strong answer covers:
- Containers are thin layers; images, build cache, volumes and logs are where the space really goes.
- Measure first with
docker system dfanddu, then clean the right thing. - Knows the difference between the prune commands, and that
-a --volumesis risky outside a build box. - Talks about prevention: log rotation, BuildKit GC, scheduled pruning, disk alerts, ephemeral runners.
The weak answer is "restart the server" or "add a bigger disk". A bigger disk only gives you more time before the same outage.
The lesson
That Friday, the build cache alone was holding more space than every container we had deleted put together. We cleaned it in five minutes, but the real fix was the log limits, the BuildKit GC setting and a nightly prune job. That runner has not filled up since.
Before you delete anything, find out what is actually using the space. Then fix the reason, not just the symptom.
What filled up your build server the last time: images, build cache, volumes or logs?
Let's connect: Free 15-min Cloud & DevOps Interview Guidance Call 👆 link in bio (topmate.io/cloudtechbydevang)
About the author
- I help engineers learn cloud, DevOps, SRE and AI in simple English, crack interviews and follow clear roadmaps. 14+ years building and running platforms on AWS, GCP, Kubernetes and Terraform. One topic a day.
Comments
Join the discussionNo comments yet. Be the first to comment.
