HallmarcDotNet

Marc Elliot Hall's Blog

Headlines

Thomas wrote about our community Saint Peters, Missouri...

Christmas card is up Check out the Flash...

Blog-o-licious We've got blogs...

Site Redesigned HallmarcDotNet has a new look...

 

Welcome to Marc's Weblog

— also known as my vanity gripe page

From sunny, Las Vegas, Nevada, this is the blog of Marc Elliot Hall, leader and system engineer extraordinaire.


January
Sun Mon Tue Wed Thu Fri Sat
     
8
 
2014
Months
Jan

Wed, 08 Jan 2014


I’ve Run out of Places to Put Stuff


Filesystem inode Trouble

When I want to find out how much space is available on a server, I use the df command, usually with a flag like -h for “human readable”. This provides me with a nice summary of space in the filesystems, in this case showing me that my root ( “/” ) filesystem has 1.8 GB available out of 9.9 GB total, for example:


	[root@localhost ~]# df -h
	Filesystem Size Used Avail Use% Mounted on
	/dev/sda2  9.9G 7.6G 1.8G  81%  /
	tmpfs      3.9G 0    3.9G   0%  /dev/shm
	/dev/sda1  485M 70M  390M  16%  /boot
	/dev/sda5  4.0G 808M 3.0G  22%  /tmp
	/dev/sda7   53G 670M  50G   2%  /var
	/dev/sda3  5.0G 216M 4.5G   5%  /var/log
	

Sometimes, things go dramatically awry, however. For example, I recently encountered a situation where ordinary users were unable to log in to a host. I could only log in as root. This is generally a bad practice (and not everybody should have the power of root anyway), so I went about troubleshooting. Among the things I did was check whether any filesystems were full with the aforementioned df -h command.

And I got this output:


	[root@localhost ~]# df -h
	df: cannot read table of mounted filesystems
	

This is suggestive of a major problem. The system is running, obviously. And, this is good: it means that the system can read at least a couple of the filesystems directly. It just can’t summarize their status for me.

So, I look at the file that is supposed to contain the table of mounted filesystems:


		[root@localhost ~]# cat /etc/mtab
	

No output at all sad

Then I look at the partition table (using fdisk -l), to see what the system thinks its disks look like:


	[root@localhost ~]# fdisk -l
	
	Disk /dev/sda: 80.5 GB, 80530636800 bytes
	255 heads, 63 sectors/track, 9790 cylinders
	Units = cylinders of 16065 * 512 = 8225280 bytes
	Sector size (logical/physical): 512 bytes / 512 bytes
	I/O size (minimum/optimal): 512 bytes / 512 bytes
	Disk identifier: 0x000d5e85
	
	   Device    Boot Start End  Blocks   Id System
	   /dev/sda1 *    1     64   512000   83 Linux
	   Partition 1 does not end on cylinder boundary.
	   /dev/sda2      64    1370 10485760 83 Linux
	   /dev/sda3      1370  2022 5242880  83 Linux
	   /dev/sda4      2022  9791 62401536  5 Extended
	   /dev/sda5      2023  2545 4194304  83 Linux
	   /dev/sda6      2545  2800 2048000  82 Linux swap / Solaris
	   /dev/sda7      2800  9791 56156160 83 Linux
	

So far so good: this system knows it has a disk (/dev/sda) with partitions (sda1 through sda7); and it at least can identify the type of filesystems they contain.

Just in case any of the fileystems aren’t mounted, using mount -a I attempt to mount them all:


	[root@localhost ~]# mount -a
	mount: /dev/sda1 already mounted or /boot busy
	mount: /dev/sda5 already mounted or /tmp busy
	mount: /dev/sda7 already mounted or /var busy
	mount: /dev/sda3 already mounted or /var/log busy
	can't create lock file /etc/mtab~1605: No space left on device (use -n flag to override)
	mount: devpts already mounted or /dev/pts busy
	mount: sysfs already mounted or /sys busy
	

That looks mostly good; they’re already showing as mounted (or just busy, but that’s a rather improbable situation). However, I see the line that says can't create lock file /etc/mtab~1605: No space left on device (use -n flag to override), which worries me. Quite a lot.

Looking a little deeper, I try to see whether /etc/mtab (my mounted file system table file) even exists at all:


	[root@localhost ~]# ls -l /etc/mt*
	-rw-r--r--. 1 root root 0 Jan 3 09:20 /etc/mtab
	

It’s there, but has zero bytes! That means the file is empty. It should contain enough information to describe the mounted filesystems — always more than zero bytes.

To summarize thus far:

  1. My users can’t log in.
  2. I have disks, which the OS has identified.
  3. I have filesystems on the disks.
  4. I have mount points for the filesystems.
  5. I probably even have mounted those filesystems.
  6. But, before I can check the status of the filesystems, I’ll have to force the system to rebuild the mounted fileystem table.

Fortunately, because Linux has a virtual filesystem containing information about the current running environment kept entirely in system RAM (the /proc filesystem), using grep and I/O redirection I can export the contents of the known mounts file ( /proc/mounts ) into a new /etc/mtab file and try my df command again:


		[root@localhost ~]# grep -v rootfs /proc/mounts > /etc/mtab
	

Now I can see that my /etc/mtab file contains 1423 bytes:


	[root@localhost ~]# ls -l /etc/mt*
	-rw-r--r--. 1 root root 1423 Jan 3 09:30 /etc/mtab
	

Then I can check whether the system can tell me about the filesystems using df and the -h flag:


	[root@localhost ~]# df -h
	Filesystem Size Used Avail Use% Mounted on
	/dev/sda2  9.9G 7.6G 1.8G  81%  /
	tmpfs      3.9G 0    3.9G   0%  /dev/shm
	/dev/sda1  485M 70M  390M  16%  /boot
	/dev/sda5  4.0G 808M 3.0G  22%  /tmp
	/dev/sda7   53G 670M  50G   2%  /var
	/dev/sda3  5.0G 216M 4.5G   5%  /var/log
	

It claims I’ve got plenty of space! Why, then, can I not use touch to create a file in the / directory, let alone log in as an ordinary user?

Possibly, because the inodes are all used up. But, “What are inodes?” you ask… Linux (and other Unix and Unix-like operating systems) in their default configuration use inodes to keep track of what file goes where on the system’s disks, and to keep metadata about the file (user and group ownership, creation time, read and write permissions, etc.). Think of inodes as the index for the file system: one inode for each file. Unfortunately, there are a finite number of inodes available (varying from filesystem to filesystem and configuration to configuration, but typically numbering well into the tens of thousands), and when they run out — even if the system has more raw space available — I can’t create any more files; thus our current problem.

Fortunately, now that my mounted filesystem table has been rebuilt, I can check for what inodes are looking like using df and the -i flag:


	[root@localhost ~]# df -ih
	Filesystem Inodes IUsed IFree IUse% Mounted on
	devtmpfs   982K   632   982K  1%    /dev
	tmpfs      985K   1     985K  1%    /dev/shm
	/dev/sda2  640K   640K  0     100%  /
	devtmpfs   982K   632   982K  1%    /dev
	/dev/sda1  126K   50    125K  1%    /boot
	/dev/sda5  256K   96K   161K  38%   /tmp
	/dev/sda7  3.4M   3.4K  3.4M  1%    /var
	/dev/sda3  320K   177   320K  1%    /var/log
	tmpfs      985K   1     985K  1%    /dev/shm
	tmpfs      985K   1     985K  1%    /dev/shm
	tmpfs      985K   1     985K  1%    /dev/shm
	

Yup, out of inodes on the / filesystem. What to do?

Join us next time for the exciting conclusion!

posted at: 16:37 |



Marc Elliot Hall St. Peters, Missouri 

Page created: 21 January 2002
Page modified: 09 December 2017

spacer About Us | Site Map | Privacy Policy | Contact Us | ©1999 - 2017 Marc Elliot Hall