No Space Left on Device, but df Shows Free Space: Inodes, Deleted Files and tmpfs

Few errors waste more time than No space left on device on a server where df -h cheerfully reports 40 percent free. The two tools are not contradicting each other. They are measuring different things, and the failure is almost always in whichever one you have not looked at yet. When we get this call, there is a short list of suspects we work through in a fixed order, because the first three account for nearly every case we see on shared and cPanel servers.

Start by asking which filesystem is actually failing

A path that looks like part of the root disk is often a separate mount. Before measuring anything, confirm which filesystem holds the file that cannot be written:

findmnt -T /var/lib/mysql
df -h /var/lib/mysql /tmp /home

If /tmp, /var or /home is its own partition, the root filesystem having plenty of space tells you nothing. Many a "mystery" ends right here, with a small /tmp partition filled by one runaway upload or a PHP session directory nobody cleaned.

Check inodes before anything else

Every file and directory consumes one inode, and on ext4 the total number is fixed when the filesystem is created. A disk can run out of inodes with most of its bytes free. This is the classic cause when the error appears while df -h looks healthy:

df -ih

Look at the IUse% column. A value at or near 100 percent is your answer. The usual offenders on a hosting server are PHP session files, mail stuck in a queue, cache plugins that write one file per page, and years of unrotated small logs. To see where the files are piling up on that mount:

find /home -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head -15

That prints the directories holding the most entries. On a large server it is slow, so run it during a quiet period. Once you know the directory, clear out what is safe to remove. Stale PHP sessions are the most common candidate, and depending on your PHP setup they live under /var/lib/php/session or as sess_* files in /tmp:

find /var/lib/php/session -type f -mtime +2 -delete

A queue that has grown into the hundreds of thousands of files deserves a closer look at why it grew. We cover the usual cause in our guide to a stuck or frozen Exim mail queue in cPanel, where a compromised script is generating the mail faster than it can leave.

Look for deleted files that are still open

Deleting a file does not free its space while a process still has it open. The name disappears from the directory, so du cannot see it, but df still counts the blocks. This is how a log file removed with rm while a service was writing to it keeps eating the disk:

lsof +L1 | head -20

Any entry with a link count of zero and a large size is space you cannot reclaim until the process closes it. Restarting the owning service releases it. If restarting is not possible right now, truncating through the process's file descriptor works, but be certain of what you are touching:

: > /proc/PID/fd/FD

The lasting fix is making sure logs are rotated with copytruncate or a post-rotate reload, so the service reopens its file instead of writing to a ghost.

Check tmpfs and reserved blocks

Two smaller causes are worth ruling out. A tmpfs mount such as /dev/shm or a size-limited /tmp has its own cap, independent of the physical disk. And ext4 reserves 5 percent of blocks for root by default, which on a large data volume can be a surprising amount of unusable space for normal users:

df -h | grep tmpfs
tune2fs -l /dev/sdb1 | grep -i "reserved block count"

On a dedicated data volume that only holds site files, lowering the reservation to 1 percent with tune2fs -m 1 /dev/sdb1 is reasonable. Leave the root filesystem's reservation alone, since it is what keeps the system usable when a disk fills.

Other things that produce the same message

If inodes, deleted files, tmpfs and reserved blocks are all clean, the remaining possibilities are less common but real. Btrfs can run out of metadata space while still reporting free data space. Containers can exhaust inodes on their own overlay layer even when the host looks fine, which docker system df will reveal. And the kernel returns this same error when a process hits its inotify watch limit, which has nothing to do with disk at all.

Make sure it does not come back

Once the server is writable again, put a number on the problem. A disk that fills every few weeks is telling you something about log rotation, backups kept locally, or a script producing files without limit. Alert on both byte usage and inode usage, because most monitoring only watches the first. Our server monitoring guide covers which checks catch this early, and the Linux maintenance checklist lists the recurring housekeeping that prevents it.

If this is happening on a production server and you would rather have an engineer trace it than guess, our Linux server management team handles exactly this kind of diagnosis, including the clean-up and the monitoring that stops a repeat.