Marc Elliot Hall's Blog


Thomas wrote about our community Saint Peters, Missouri...

Christmas card is up Check out the Flash...

Blog-o-licious We've got blogs...

Site Redesigned HallmarcDotNet has a new look...


Welcome to Marc's Weblog

— also known as my vanity gripe page

From sunny, Las Vegas, Nevada, this is the blog of Marc Elliot Hall, leader and system engineer extraordinaire.

Sun Mon Tue Wed Thu Fri Sat

Thu, 26 Nov 2015

Left Holding (Open) the Bag

Filesystem inode Trouble Part II

You may recall that in January, 2014 I wrote a piece about inodes and filesystems behaving badly. At the behest of a colleague (Hi, Josh!), this is the exciting conclusion to that saga.

When last we left the action, my free disk space looked like this:

	[root@localhost ~]# df -h
	Filesystem Size Used Avail Use% Mounted on
	/dev/sda2  9.9G 7.6G 1.8G  81%  /
	tmpfs      3.9G 0    3.9G   0%  /dev/shm
	/dev/sda1  485M 70M  390M  16%  /boot
	/dev/sda5  4.0G 808M 3.0G  22%  /tmp
	/dev/sda7   53G 670M  50G   2%  /var
	/dev/sda3  5.0G 216M 4.5G   5%  /var/log

Yet, ordinary users were unable to log in, and I could not create new files on the root (“/” ) filesystem.

To summarize thus far:

  1. My users can’t log in.
  2. I have disks, which the OS has identified.
  3. I have filesystems on the disks.
  4. I have mount points for the filesystems.
  5. I probably even have mounted those filesystems.
  6. I was able to rebuild the mounted fileystem table.

After digging about and recreating the filesystem description table, I determined that the system had run out of inodes:

	[root@localhost ~]# df -ih
	Filesystem Inodes IUsed IFree IUse% Mounted on
	devtmpfs   982K   632   982K  1%    /dev
	tmpfs      985K   1     985K  1%    /dev/shm
	/dev/sda2  640K   640K  0     100%  /
	devtmpfs   982K   632   982K  1%    /dev
	/dev/sda1  126K   50    125K  1%    /boot
	/dev/sda5  256K   96K   161K  38%   /tmp
	/dev/sda7  3.4M   3.4K  3.4M  1%    /var
	/dev/sda3  320K   177   320K  1%    /var/log
	tmpfs      985K   1     985K  1%    /dev/shm
	tmpfs      985K   1     985K  1%    /dev/shm
	tmpfs      985K   1     985K  1%    /dev/shm

A quick reminder about inodes: Linux (and other Unix and Unix-like operating systems) in their default configuration use inodes to keep track of what file goes where on the system’s disks, and to keep metadata about the file (user and group ownership, creation time, read and write permissions, etc.). Think of inodes as the index for the file system: at least one inode for each file (you will have more than one inode per file if the file is linked to multiple locations in your filesystem). Unfortunately, there are a finite number of inodes available (varying from filesystem to filesystem and configuration to configuration, but typically numbering well into the tens of thousands), and when they run out — even if the system has more raw space available — it can’t create any more files. Moreover, the number of inodes a filesystem has cannot be (easily) changed after it is created.

Fortunately, there is a simple solution!

Unfortunately, I no longer have access to the system that I was troubleshooting when I wrote my earlier post. However, the fix is pretty universal. With a little script-fu, we can find out how many files each directory in the filesystem has. Once we have identified the quantity and location, we can determine whether there is any particular reason to keep those files. Most of the time in a situation like this, some runaway process has been spewing data that doesn’t get cleaned up properly, either because the process never terminates or because the process isn’t properly coordinating with tools like logrotate. If the data being spewed is a bunch of small files, we can then simply delete the files.

To start, then:

	echo 'echo $(ls -a "${1}" | wc -l) ${1}' > /tmp/files_`date +%F`
	chmod 700 /tmp/files_`date +%F`
	find . -mount -type d -print0 | xargs -0 -n1 /tmp/files_`date +%F` | sort -n | tail -10

This will:

  1. Generate a list of directories in the root filesystem;
  2. Count the number of files in each directory;
  3. Spit out the list with two columns:
    • the right column with the directory name,
    • the left column with the file count;
  4. Sort the two-column list by the file count value;
  5. If the list is more than 10 lines, only show you the 10 with the most files.

Usually, the directory with the most files is your culprit. You’ll want to verify that, then determine whether you should just delete the oldest umpteen files in that directory, all the files in the directory, or whatever other subset is appropriate. You’ll also want to correct whatever process is generating the files. Short of rewriting the program that spewed so much data, you can fix this a number of ways. Three straightforward methods are:

  1. Set up a wrapper script that manages it properly,
  2. Create a logrotate script to clean it up, or
  3. Build a cron job that will periodically do that for you.

Don’t forget to delete your temporary file:

	rm -f /tmp/files_`date +%F`

Happy hunting!

posted at: 20:20 |

Marc Elliot Hall St. Peters, Missouri 

Page created: 21 January 2002
Page modified: 09 December 2017

spacer About Us | Site Map | Privacy Policy | Contact Us | ©1999 - 2017 Marc Elliot Hall