Blog
4 min read

"No Space Left on Device": Finding and Freeing Disk Space on a Server

When a server's disk fills up, databases stop writing, deploys fail and apps crash with ENOSPC. How to find what's using space with df and du, the usual culprits (logs, Docker, journald, old releases, deleted-but-open files), inode exhaustion, and how to stop it happening again.

Error: ENOSPC: no space left on device, write

could not extend file "base/16384/2619": No space left on device

A full disk is one of the most common — and most avoidable — causes of production outages. When it happens, everything that writes fails at once: the database, logs, uploads, npm install, even SSH logins sometimes. Here's how to find the space quickly and stop it happening again.

Step 1: Confirm which disk is full

df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        40G   40G     0 100% /
/dev/sdb        100G   12G   88G  12% /mnt/data

Look for Use% near 100% and note the mount point. Also check inodes — the number of files, which can run out even when there's space:

df -i

If IUse% is 100%, the problem is millions of tiny files (session files, cache entries, mail queues), not size.

Step 2: Find what's using it

Walk down from the full mount, biggest first:

sudo du -xh / --max-depth=1 2>/dev/null | sort -h | tail -15
sudo du -xh /var --max-depth=1 2>/dev/null | sort -h | tail -15

-x stays on one filesystem. Repeat into the biggest directory until you find the culprit. ncdu (sudo apt install ncdu, then sudo ncdu -x /) does this interactively and is much faster to navigate.

The usual culprits

Application logs. An app logging every request to a file that never rotates. Check /var/log, your app directory, and PM2's logs (~/.pm2/logs). (Structured logging)

journald. The system journal can grow to gigabytes:

journalctl --disk-usage
sudo journalctl --vacuum-size=500M

Cap it permanently with SystemMaxUse=500M in /etc/systemd/journald.conf.

Docker. Images, stopped containers, build cache and volumes accumulate with every deploy:

docker system df                 # what Docker is using
docker system prune              # stopped containers, unused networks, dangling images, build cache
docker image prune -a            # all images not used by a container (careful)

Be cautious with docker volume prune — volumes hold data (like your database). Container log files under /var/lib/docker/containers/ also grow without limit unless you configure log rotation (max-size in daemon.json). (Docker volumes vs bind mounts)

Old releases and build artifacts. Release folders, node_modules copies, .next caches from every deploy. Keep the last few releases only.

Database growth. Postgres WAL files piling up (often because of a stuck replication slot or failing archive command), or table bloat. Check pg_wal size and SELECT * FROM pg_replication_slots;. (Postgres VACUUM and bloat, point-in-time recovery)

Backups stored on the same disk. A nightly dump with no cleanup will eventually fill any disk — and doesn't protect you if the disk dies anyway. Move backups off-server. (Backups for beginners)

Uploads. User files on local disk grow forever. (What is object storage?)

Package caches. sudo apt clean, npm cache clean --force, ~/.cache.

"I deleted files but the space didn't come back"

If a running process still has a deleted file open (classic: you deleted a huge log file the app is still writing to), the space isn't freed until the process closes it.

sudo lsof +L1     # open files with zero links (deleted)

Restart the process holding it — or, to free space without restarting, truncate the file instead of deleting it next time:

: > /var/log/myapp/huge.log    # empties it in place

Emergency space

If the disk is so full you can't even work, free a little quickly: vacuum journald, apt clean, prune Docker build cache, or truncate one oversized log. Then investigate properly.

On ext4, 5% of the disk is reserved for root by default — which is why root can sometimes still write when other users can't.

Prevent it

  • Rotate logs. logrotate for files, max-size/max-file for Docker, SystemMaxUse for journald.
  • Alert at 80%. Disk usage is the easiest metric to monitor and the most embarrassing to miss. (Know when your app is down)
  • Clean up deploys. Prune old images and releases as part of the deploy script.
  • Keep data elsewhere. Uploads in object storage, backups off-server.
  • Size the disk with headroom, or use a separate volume for data you can grow independently.

The summary

  • df -h (and df -i for inodes) shows what's full; du or ncdu shows why.
  • Usual suspects: logs, journald, Docker, old releases, WAL, local backups, uploads.
  • Deleted-but-open files hold space until the process restarts — truncate instead.
  • Rotate logs, prune on deploy, alert at 80%, keep data and backups off the root disk.

EasySpawn servers come with log rotation, disk monitoring and off-server backups handled, plus bucket storage for uploads — so a full disk doesn't take your app down at 3 a.m. See how it works or join the waitlist.

Related: How to Secure a New VPS · Structured Logging · Reduce Docker Image Size · Postgres VACUUM and Table Bloat

Keep reading