I’ll admit it. I’ve been neglecting this blog. Well, more accurately, I’ve been neglecting the sysadmin portion of the site. I haven’t updated anything, much less rebooted the server that is feeding you this page, amongst other things. The only thing I’ve been doing is periodically logging in to renew my letsencrypt cert. So about 2 minutes combined over the last 7 months?
I hopped on today to check for uptime and install Ubuntu patches and lo and behold I was met with some errors. I simply ran
and things were looking good. 230+ items to update. Seems about right for 330 days of uptime and not patching (shh I know it’s terrible). I ran
to install patches but that wasn’t so kind. I was met with a bunch of output indicating that I needed to install a newer kernel in order to satisfy some dependencies. Why it wasn’t installing? I wasn’t sure at first. The output told me to attempt running this command to rectify:
apt-get -f install
This too failed but I received more interesting output:
No apport report written because the error message indicates a disk full error
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Interesting since nothing else indicated that the filesystem was full. I did some more investigating using
du -sh and
df and my filesystem was a mere 67% full. What gives? I’ll tell ya: inodes.
For those who do not understand the concept of inodes let’s break it down simply using a file cabinet as an analogy:
File Cabinet – File System
Drawers – Filenames
Folders – inodes
Documents – File Data
It’s important to distinguish the difference between filenames and inode data. The inode data is the metadata associated with data on the filesystem. This metadata includes access and modification dates, file size, permissions, etc. A common use case that better explains why inode data is separate from the filename is the usage of hard links. Say you have a regular file. It has a filename, the associated inode metadata, and the actual data. Creating a hard link to the file is really just creating another filename that points to the inode data. Check out the below screenshot:
So what’s going on here? I created “file1” using touch. Running the
ls command gives me a listing of filenames, which returns “file1” as expected. Running
ls -l returns inode data, hence why the owner, permissions, etc data is shown. I then created a hard link, named “file1link” which links to “file1”. Running
ls -l again shows me the inode data for each of the filenames. This inode data is the same for the two filenames. Running
ls -i shows me the inode number for each of the file names. They are the same for both filenames because they are referencing the same inode data. Make sense?
This is all cool stuff, but how does this play into the error I was seeing? Quite simple: I was out of inodes! I read that upon filesystem creation the ratio of inodes to disk space is 16Kb per inode so as long as your average file size is above 16Kb you shouldn’t run out of inodes. Apparently I have a shit ton of small files eating up my inodes without taking up enough space to actual fill my filesystem. Interesting little thing going on here. I ran
df -i to get some inode data and dun dun dun 100% full! No more inodes left. I took this screenshot after I did my cleaning up but this is what
df -i returns:
I did some more digging on the Internet and found that removing old kernels from my system should alleviate the problem I was having. The kernel I needed was queued up to install but I found that I had a dozen or so kernels still lingering around. Perhaps these extra kernels were eating up my inodes? Here’s the sequence I used to free up some inodes by forcefully deleting some extraneous kernels.
First I ran the following command to list my currently booted kernel:
The below returns a list of all kernels on my system except the one I am booted to:
dpkg -l | tail -n +6 | grep -E 'linux-image-[0-9]+' | grep -Fv $(uname -r)
Yea some of those can go, so let’s get to it.
linux-image-4.4.0-51-generic is the oldest on here, so let’s zap that one. Use the following command to remove the initrd.img file (this is due to Bug 1678187).
sudo update-initramfs -d -k 4.2.0-51-generic
We now need to use dpkg to finalize removal:
sudo dpkg --purge linux-image-4.2.0-51-generic linux-image-extra-4.2.0-51-generic
sudo dpkg --purge linux-headers-4.2.0-51-generic
sudo dpkg --purge linux-headers-4.2.0-51
It is possible that the first dpkg command fails due to something being dependent upon that particular kernel. If this happens dpkg will alert you and you’ll need to take some action.
That should do it. I removed 2 kernels and freed up (as indicated above) over 60,000 inodes. I was then able to successfully update Ubuntu with no further issues.
I’m just a normal sysadmin type guy who likes cybersecurity a lot.