HallmarcDotNet

Marc Elliot Hall's Blog

Headlines

Thomas wrote about our community Saint Peters, Missouri...

Christmas card is up Check out the Flash...

Blog-o-licious We've got blogs...

Site Redesigned HallmarcDotNet has a new look...

 

Welcome to Marc's Weblog

— also known as my vanity gripe page

From sunny, Las Vegas, Nevada, this is the blog of Marc Elliot Hall, leader and system engineer extraordinaire.


January
Sun Mon Tue Wed Thu Fri Sat
     
8
 
2014
Months
Jan

Wed, 08 Jan 2014


I’ve Run out of Places to Put Stuff


Filesystem inode Trouble

When I want to find out how much space is available on a server, I use the df command, usually with a flag like -h for “human readable”. This provides me with a nice summary of space in the filesystems, in this case showing me that my root ( “/” ) filesystem has 1.8 GB available out of 9.9 GB total, for example:

[root@localhost ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 9.9G 7.6G 1.8G 81% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sda1 485M 70M 390M 16% /boot
/dev/sda5 4.0G 808M 3.0G 22% /tmp
/dev/sda7 53G 670M 50G 2% /var
/dev/sda3 5.0G 216M 4.5G 5% /var/log

Sometimes, things go dramatically awry, however. For example, I recently encountered a situation where ordinary users were unable to log in to a host. I could only log in as root. This is generally a bad practice (and not everybody should have the power of root anyway), so I went about troubleshooting. Among the things I did was check whether any filesystems were full with the aforementioned df -h command.

And I got this output:

[root@localhost ~]# df -h
df: cannot read table of mounted filesystems

This is suggestive of a major problem. The system is running, obviously. And, this is good: it means that the system can read at least a couple of the filesystems directly. It just can’t summarize their status for me.

So, I look at the file that is supposed to contain the table of mounted filesystems:

[root@localhost ~]# cat /etc/mtab

No output at all sad

Then I look at the partition table (using fdisk -l), to see what the system thinks its disks look like:

[root@localhost ~]# fdisk -l

Disk /dev/sda: 80.5 GB, 80530636800 bytes
255 heads, 63 sectors/track, 9790 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000d5e85

Device Boot Start End Blocks Id System
/dev/sda1 * 1 64 512000 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 64 1370 10485760 83 Linux
/dev/sda3 1370 2022 5242880 83 Linux
/dev/sda4 2022 9791 62401536 5 Extended
/dev/sda5 2023 2545 4194304 83 Linux
/dev/sda6 2545 2800 2048000 82 Linux swap / Solaris
/dev/sda7 2800 9791 56156160 83 Linux

So far so good: this system knows it has a disk (/dev/sda) with partitions (sda1 through sda7); and it at least can identify the type of filesystems they contain.

Just in case any of the fileystems aren’t mounted, using mount -a I attempt to mount them all:

[root@localhost ~]# mount -a
mount: /dev/sda1 already mounted or /boot busy
mount: /dev/sda5 already mounted or /tmp busy
mount: /dev/sda7 already mounted or /var busy
mount: /dev/sda3 already mounted or /var/log busy
can't create lock file /etc/mtab~1605: No space left on device (use -n flag to override)
mount: devpts already mounted or /dev/pts busy
mount: sysfs already mounted or /sys busy

That looks mostly good; they’re already showing as mounted (or just busy, but that’s a rather improbable situation). However, I see the line that says can't create lock file /etc/mtab~1605: No space left on device (use -n flag to override), which worries me. Quite a lot.

Looking a little deeper, I try to see whether /etc/mtab (my mounted file system table file) even exists at all:

[root@localhost ~]# ls -l /etc/mt*
-rw-r--r--. 1 root root 0 Jan 3 09:20 /etc/mtab

It’s there, but has zero bytes! That means the file is empty. It should contain enough information to describe the mounted filesystems — always more than zero bytes.

To summarize thus far:

  1. My users can’t log in.
  2. I have disks, which the OS has identified.
  3. I have filesystems on the disks.
  4. I have mount points for the filesystems.
  5. I probably even have mounted those filesystems.
  6. But, before I can check the status of the filesystems, I’ll have to force the system to rebuild the mounted fileystem table.

Fortunately, because Linux has a virtual filesystem containing information about the current running environment kept entirely in system RAM (the /proc filesystem), using grep and I/O redirection I can export the contents of the known mounts file ( /proc/mounts ) into a new /etc/mtab file and try my df command again:

[root@localhost ~]# grep -v rootfs /proc/mounts > /etc/mtab

Now I can see that my /etc/mtab file contains 1423 bytes:

[root@localhost ~]# ls -l /etc/mt*
-rw-r--r--. 1 root root 1423 Jan 3 09:30 /etc/mtab

Then I can check whether the system can tell me about the filesystems using df and the -h flag:

[root@localhost ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 9.9G 7.6G 1.8G 81% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sda1 485M 70M 390M 16% /boot
/dev/sda5 4.0G 808M 3.0G 22% /tmp
/dev/sda7 53G 670M 50G 2% /var
/dev/sda3 5.0G 216M 4.5G 5% /var/log

It claims I’ve got plenty of space! Why, then, can I not use touch to create a file in the / directory, let alone log in as an ordinary user?

Possibly, because the inodes are all used up. But, “What are inodes?” you ask… Linux (and other Unix and Unix-like operating systems) in their default configuration use inodes to keep track of what file goes where on the system’s disks, and to keep metadata about the file (user and group ownership, creation time, read and write permissions, etc.). Think of inodes as the index for the file system: one inode for each file. Unfortunately, there are a finite number of inodes available (varying from filesystem to filesystem and configuration to configuration, but typically numbering well into the tens of thousands), and when they run out — even if the system has more raw space available — I can’t create any more files; thus our current problem.

Fortunately, now that my mounted filesystem table has been rebuilt, I can check for what inodes are looking like using df and the -i flag:

[root@localhost ~]# df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
devtmpfs 982K 632 982K 1% /dev
tmpfs 985K 1 985K 1% /dev/shm
/dev/sda2 640K 640K 0 100% /
devtmpfs 982K 632 982K 1% /dev
/dev/sda1 126K 50 125K 1% /boot
/dev/sda5 256K 96K 161K 38% /tmp
/dev/sda7 3.4M 3.4K 3.4M 1% /var
/dev/sda3 320K 177 320K 1% /var/log
tmpfs 985K 1 985K 1% /dev/shm
tmpfs 985K 1 985K 1% /dev/shm
tmpfs 985K 1 985K 1% /dev/shm

Yup, out of inodes on the / filesystem. What to do?

Join us next time for the exciting conclusion!

posted at: 16:37 |



Marc Elliot Hall St. Peters, Missouri 

Page created: 21 January 2002
Page modified: 09 December 2017

spacer About Us | Site Map | Privacy Policy | Contact Us | ©1999 - 2017 Marc Elliot Hall