I Kept Deleting Logs for 48 Hours. The Inodes Were Already Gone.

Have you ever watched a two-kilobyte write fail with No space left on device while df -h still showed free gigabytes? I did, and I spent the next forty-eight hours cleaning the wrong evidence. This is the reconstructed field notebook from that session, including the commands I ran, the ones that misled me, and the checklist I now run before I blame the disk. Nothing here is a benchmark, a quota promise, or a claim about hardware I did not measure. I was iterating on a small Python worker that dumped JSON sidecars next to each run. The worker itself was unremarkable. The failure mode was not. Hour 0: the write that should have been boring The first traceback looked like a disk problem, so I treated it like a disk problem. Would you have done anything else with ENOSPC staring at you from a three-line stack? I would not, and that is exactly how the next two days started. OSError: [Errno 28] No space left on device: 'runs/2026-09-05T07-12-04.json' I ran the obvious command, got a comforting number, and closed the wrong investigation. df -h reported plenty of space on the root filesystem, and /tmp looked equally relaxed. I even created a dummy file in $HOME by hand, which succeeded, so I told myself the worker path was special. df -h df -h /tmp /var /home touch ~/probe-ok.txt && ls -l ~/probe-ok.txt That last touch was the trap. Can a filesystem accept a file in one directory and refuse a tiny file in another while still having blocks to spare? Yes, and inode exhaustion is the boring reason. I did not ask that question for twelve hours. What I tried first, and why it felt reasonable I treated the symptom as log rot, because that is the story operators tell each other. I truncated worker logs, deleted old JSON sidecars I could see, and reran the job with a smaller batch. The write still failed, sometimes on file number twenty, sometimes on file number four. Truncated worker.log and debug.log with : > file instead of deleting the path. Removed a handful of large .jsonl fil