# You Deleted the Docker Containers, but the Disk Is Still Full: Why, and How to Fix It

It was a Friday evening. Every pipeline on our shared CI runner turned red at the same time, all with the same message:

```
no space left on device
```

The team had already done the "obvious" thing. "We deleted all the old containers," someone said in the channel. I logged in and ran `df -h`. The disk was still 98% full.

This is one of the most common Docker problems I see on build servers, and in 14+ years of working with infrastructure it still catches good engineers. It is also a question I like to ask in DevOps interviews, because the answer shows whether someone understands how Docker actually stores things.

## Why deleting containers doesn't free much space

A container is a thin, writable layer on top of an image. When you delete a container, you delete that thin layer, and usually not much else.

The space on a Docker host is mostly used by four other things:

1. **Images.** Every build creates a new image. Old tags and untagged ("dangling") images stay on the disk until you remove them. Deleting a container does not delete the image it came from.
2. **Build cache.** BuildKit keeps intermediate layers so the next build is fast. On a busy CI runner that builds many projects, this cache can grow to tens of gigabytes.
3. **Volumes.** Named and anonymous volumes survive after the container is gone. Databases and test containers often leave them behind.
4. **Container logs.** With the default `json-file` logging driver and no size limit, one noisy container can write logs until the disk is full.

So "I deleted the containers" is like taking the cups off the table and leaving the kitchen full of dirty dishes.

## Step 1: Find out what is using the space

Don't start deleting blindly. First, ask Docker:

```bash
docker system df
```

You'll see something like this:

```
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          187       6         61.2GB    58.9GB (96%)
Containers      6         4         310MB     120MB (38%)
Local Volumes   42        3         9.8GB     9.1GB (92%)
Build Cache     1240      0         38.4GB    38.4GB
```

(These numbers are an illustration of what a busy runner typically looks like.) The `RECLAIMABLE` column tells you where the easy wins are.

For more detail:

```bash
docker system df -v          # per image, volume and cache entry
du -sh /var/lib/docker/*     # what is on disk, by Docker's own folders
```

If `docker system df` looks small but the disk is still full, check the logs:

```bash
sudo du -sh /var/lib/docker/containers/*/*-json.log | sort -h | tail
```

## Step 2: Clean up on purpose

Each cleanup command has a clear scope. Use the one that matches what you found.

```bash
# Remove dangling (untagged) images
docker image prune

# Remove all images not used by a container (bigger cleanup)
docker image prune -a

# Remove build cache older than 3 days
docker builder prune --filter until=72h

# Remove volumes not used by any container
docker volume prune

# Everything unused at once - build servers only
docker system prune -a --volumes
```

A word of warning: `docker system prune -a --volumes` is fine on a CI runner where everything can be rebuilt. On a server that runs real workloads, it can delete a volume that holds data you care about. Always read the confirmation prompt.

For a huge log file, don't just `rm` it while the container is running; the process keeps the file open and the space is not freed. Truncate it instead:

```bash
sudo truncate -s 0 /var/lib/docker/containers/<id>/<id>-json.log
```

## Step 3: Stop it from happening again

Cleaning up once is not a fix. These are the changes I make on every build host.

**1. Limit log size** in `/etc/docker/daemon.json`:

```json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "3"
  }
}
```

Restart Docker after the change. New containers pick it up; old ones need to be recreated.

**2. Cap the build cache.** With BuildKit you can set a garbage-collection limit so the cache cleans itself:

```json
{
  "builder": {
    "gc": {
      "enabled": true,
      "defaultKeepStorage": "20GB"
    }
  }
}
```

**3. Schedule a prune job** on CI runners, for example a nightly cron or systemd timer:

```bash
docker image prune -a -f --filter until=168h
docker builder prune -f --filter until=72h
```

**4. Alert early.** Set a disk alert at 80%, not 95%. At 95% you are already firefighting on a Friday evening.

**5. Use ephemeral runners where you can.** Auto-scaling or container-based runners that are thrown away after each job never collect old images in the first place.

## The interview angle

When I ask "You removed the containers, but the disk is still full. Why?", here's what a strong answer covers:

- Containers are thin layers; **images, build cache, volumes and logs** are where the space really goes.
- **Measure first** with `docker system df` and `du`, then clean the right thing.
- Knows the **difference between the prune commands**, and that `-a --volumes` is risky outside a build box.
- Talks about **prevention**: log rotation, BuildKit GC, scheduled pruning, disk alerts, ephemeral runners.

The weak answer is "restart the server" or "add a bigger disk". A bigger disk only gives you more time before the same outage.

## The lesson

That Friday, the build cache alone was holding more space than every container we had deleted put together. We cleaned it in five minutes, but the real fix was the log limits, the BuildKit GC setting and a nightly prune job. That runner has not filled up since.

Before you delete anything, find out what is actually using the space. Then fix the reason, not just the symptom.

What filled up your build server the last time: images, build cache, volumes or logs?

---

> **Let's connect: Free 15-min Cloud & DevOps Interview Guidance Call 👆 link in bio**
> ([topmate.io/cloudtechbydevang](https://topmate.io/cloudtechbydevang/2358399))

