Today’s sysadmin tip: What’s filling my /tmp?

So, you’ve just been paged eh? And the disk is full? Again? Didn’t check those munin graphs, did you? And now, everything burns, and you have to clear some space on /tmp that you forgot to move away from your root filesystem? Now: There are no large files in /tmp or anywhere else, but the disk is certainly full. Remember you heard something about deleted files that are being kept open? lsof may tell right? Ah, yes, there they are, but which one is eating my disk? Halp!

here’s a little script I called invdu – that is invisible du, that may help. It takes one or more search patterns, typically a directory, finds the correct fd link under /proc for those deleted files, and gives you the typical du output. For example:

# invdu -m /tmp | sort -n
0 /tmp/ib5Bz0TA
0 /tmp/ibceNUbK
2556 /tmp/ibxNOnKl

Aha!

#lsof | grep /tmp/ibxNOnKl
mysqld 8085 mysql 11u REG 8,3 2677165648 163873 /tmp/ibxNOnKl (deleted)

mysql using an undersized /tmp for that? Shame on you. Now, extinguish that fire, and return to that beer|nap|tvshow you just planned to take.

Leave a Reply