Latest Tweets

Whatever happened to my free disk space?

The issue

A GNU/Linux box running Debian Lenny EM64T reported a total disk usage for the slash partition of 13GB. However, running a du command the total amount of  space occupied  – not including those external mount points accessible from the root device -, was almost half that size. We used lsof in order to determine whether there was any running process accessing certain deleted files, but it turned out that was not the case. We decided to force a file system check, but the problem was still there. Below, the total space used reported by  df utility, which call statfs(), and our trivial command relying on du utility, showing clearly something was not exactly right there:

Filesystem            Size  Used Avail Use% Mounted onLog Out
/dev/sda1              14G   13G  964M  93% /

antares:/# totalsizei=0;for d in *; do if [ $d != “home” -a $d != “proc” -a $d != “mnt” -a $d != “media” -a $d != “sys” ]; then sizel=`du -s /$d 2>/dev/null`; if [ ! -z “$sizel” ]; then sizei=`echo $sizel|cut -d” ” -f1`; totalsizei=`expr $totalsizei + $sizei`; echo “Directory: $d , $sizei Kbytes”; fi; fi; done; echo “Total: $totalsizei Kbytes”

Directory: bin , 4688 Kbytes
Directory: boot , 30064 Kbytes
Directory: cdrom , 0 Kbytes
Directory: dev , 852 Kbytes
Directory: emul , 135584 Kbytes
Directory: etc , 27948 Kbytes
Directory: initrd , 4 Kbytes
Directory: initrd.img , 0 Kbytes
Directory: initrd.img.old , 0 Kbytes
Directory: lib , 165180 Kbytes
Directory: lib32 , 0 Kbytes
Directory: lib64 , 0 Kbytes
Directory: lost+found , 16 Kbytes
Directory: opt , 846924 Kbytes
Directory: prova , 40 Kbytes
Directory: root , 64232 Kbytes
Directory: sbin , 3780 Kbytes
Directory: selinux , 4 Kbytes
Directory: srv , 4 Kbytes
Directory: tmp , 37424 Kbytes
Directory: usr , 5192540 Kbytes
Directory: var , 505708 Kbytes
Directory: vmlinuz , 0 Kbytes
Directory: vmlinuz.old , 0 Kbytes
Total: 7014992 Kbytes

Thus, it was clear enough, according to that last output, that the actual total amount of disk usage was about 7GB, instead of 13GB!

Analysing the file-system

Using dumpe2fs, we got the total number of free blocks and the block size for the slash partition. This information is stored in the super-block, so we decided to have a look at the different super-block copies, just in case the first one was somehow inconsistent. To get a complete list of the super-block copies, we ran testdisk:

Disk /dev/sda – 1500 GB / 1397 GiB – CHS 182401 255 63
Partition                  Start        End    Size in sectors
Linux                    0   1  1  1823 254 62   29302496
superblock 0, blocksize=4096
superblock 32768, blocksize=4096
superblock 98304, blocksize=4096
superblock 163840, blocksize=4096
superblock 229376, blocksize=4096
superblock 294912, blocksize=4096
superblock 819200, blocksize=4096
superblock 884736, blocksize=4096
superblock 1605632, blocksize=4096
superblock 2654208, blocksize=4096

So, we checked every single copy of those super-blocks but all of them seemed consistent. Dumpe2fs told us we had a total of 431293 out of 3662812 blocks available. The block size for this partition was 4096. Apart from the reserved blocks, which was 183140, it did not seem feasible to find out where the heck the rest of the space was wasted:

dumpe2fs -h /dev/sda1

Block count:              3662812
Reserved block count:     183140
Free blocks:              431293

Searching for file holes

Another approach involved file holes. It is quite simple to create a file with holes in it. For instance:

echo -ne “X” |dd of=test_holes  bs=4096 seek=6

If we have a look at that recently created file, we see it has a total size of 25KB, but, in fact, it is a single byte (“X”) followed by (4096*6)-1 null characters. So, the GNU/Linux Kernel is not going to allocate those 5 next blocks just to store null characters right away. Therefore, we’ve got a file hole. Let’s check that out:

du test_holes
4    test_holes

du –apparent-size test_holes
25    test_holes

During the first du execution, we’ve got a 4KB file size. That is because we’ve got just a single “X” character, 1 byte, and the block size for that partition is 4096 bytes. Therefore, the GNU/Linux Kernel has allocated just 1 block for the test_holes file. That is, it does occupy 4KB. This block will contain that “X” character. A single directed block will do:

Inode: 3195202   Type: regular    Mode:  0644   Flags: 0x0
Generation: 226257934    Version: 0x00000000
User:   501   Group:  1000   Size: 24577
File ACL: 0    Directory ACL: 0
Links: 1   Blockcount: 8
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x4e4df313 — Fri Aug 19 07:22:27 2011
atime: 0x4e4df313 — Fri Aug 19 07:22:27 2011
mtime: 0x4e4df313 — Fri Aug 19 07:22:27 2011
BLOCKS:
(6):6391821
TOTAL: 1

But running du again with the –apparent-size flag (second output above), the size is quite different: that is, 25KB. According to the du man-page, this is the way to find file holes. Those 25KB would be, apparently, the test_holes real size, from a user’s point of view, and so the command ls says:

ls -l test_holes
-rw-r–r– 1 tonicas tonicas 24577 Aug 19 07:22 test_holes

Unluckily, those file holes are not reported by statfs(). Thus, running df command as shown early, all the holes present in the entire file-system are not going to be accounted. We can demonstrate that right now:

/dev/sda6              95G   51G   44G  54% /home
echo -ne “X” | dd of=test1GB bs=1024 seek=100000

ls -lh test1GB
-rw-r–r– 1 root root 977M Aug 19 07:38 test1GB
/dev/sda6              95G   51G   44G  54% /home

As you can see, our huge hole is not affecting the total space used on our hard disk, which is still 95GB, and not 96GB. Well, to be precise, it is affecting it, but just because of that real used space, without holes. Despite so, we ran our command relying on du in order to find out if we had holes over there. This time we decided to include all the special directories in the slash partition, even the pseudo file-system /proc. Okay, we do know /proc does not occupy space on the disk, but even so, we included it. Call it a hunch.

Directory: bin , 4422 Kbytes
Directory: boot , 29937 Kbytes
Directory: cdrom , 1 Kbytes
Directory: dev , 63 Kbytes
Directory: emul , 133043 Kbytes
Directory: etc , 23475 Kbytes
Directory: initrd , 4 Kbytes
Directory: initrd.img , 1 Kbytes
Directory: initrd.img.old , 1 Kbytes
Directory: lib , 154690 Kbytes
Directory: lib32 , 1 Kbytes
Directory: lib64 , 1 Kbytes
Directory: lost+found , 16 Kbytes
Directory: media , 21 Kbytes
Directory: mnt , 4 Kbytes
Directory: opt , 822389 Kbytes
Directory: proc , 5177622 Kbytes
Directory: prova , 40 Kbytes
Directory: root , 62840 Kbytes
Directory: sbin , 3510 Kbytes
Directory: selinux , 4 Kbytes
Directory: srv , 4 Kbytes
Directory: sys , 591325 Kbytes
Directory: tmp , 27704 Kbytes
Directory: usr , 4734477 Kbytes
Directory: var , 501626 Kbytes
Directory: vmlinuz , 1 Kbytes
Directory: vmlinuz.old , 1 Kbytes
Total: 12267223 Kbytes

Okay; clearly the pseudo file-system /proc was about 5,1GB. No way it could be the culprit. We all know /proc is just a pseudo file-system, it does not occupy hard disk space! But still, there it was: the only feasible explanation about where the heck  those missing Giga bytes could be.

/proc/kcore

On modern GNU/Linux kernels, /proc/kcore is just a file representation of the whole memory the kernel can address. For example, if we have a 5GB /proc/kcore file, that means our GNU/Linux box can address a total amount of 5GBof memory. Obviously, this file is not a real one, because it is in the /proc/ pseudo file-system and , therefore, it does not waste hard disk space. In our precise issue, that GNU/Linux box had a 5GB /proc/kcore file, thus the reason why du –apparent-size over /proc reported such a huge amount of hypothetically used space:

ls -lh /proc/kcore
-r——– 1 root root 5.0G 2011-08-19 08:30 /proc/kcore

So we sort of thought some bug was involved with the statfs() call, including, somehow, the /proc/kcore in the accounting for the total hard disk space occupied. Almost!

If that was so, we had to check whether that /proc/kcore was involved. We looked for its i-node number and then we used debugfs to check it out:

ls -lhi /proc/kcore
4026531863 -r——– 1 root root 5.0G 2011-08-19 08:33 /proc/kcore
antares:/# debugfs /dev/sda1
debugfs 1.41.3 (12-Oct-2008)
debugfs:  ncheck 4026531863
Inode    Pathname

It did make sense, obviously. That i-node was nowhere to be seen because it did not exist! But we insisted:

debugfs:  cd /proc

debugfs:  ls

623579  (16) kcore 623580  (20) key-users    623581  (12) kmsg

What? Another /proc/kcore on that partition with a different i-node number? It couldn’t be! So we decided to have a look at that new i-node number more carefully:

debugfs:  ncheck 623579
Inode    Pathname
623579    /proc/kcore

debugfs:  stat kcore
Inode: 623579   Type: regular    Mode:  0400   Flags: 0x0
Generation: 2774104123    Version: 0x00000000
User:     0   Group:     0   Size: 5301604352
File ACL: 0    Directory ACL: 0
Links: 1   Blockcount: 10364832
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x4ce3a3cc — Wed Nov 17 10:43:40 2010
atime: 0x4ce3a3cc — Wed Nov 17 10:43:40 2010
mtime: 0x4ce39fa4 — Wed Nov 17 10:25:56 2010
Size of extra inode fields: 4

Now, things did make sense. We had another /proc/kcore file, and therefore, another entire /proc directory present on our /dev/sda1 partition. This entire /proc physical directory was  in fact occupying hard disk space, but it was not visible using standard commands like du, ls, etc, because our actual pseudo file-system /proc was obviously mounted there. It was so evident because of the i-node information we got after running debugfs over /proc/kcore. Look at the times!

To confirm that theory, we imaged the /dev/sda1 running clone2fs. The entire image file was about 13GB in size. And in there we got a /proc directory with actual files in it, including a huge 5GB kcore file.

Fixing /proc

This issue is yet to be fixed because though we can delete /proc/kcore using debugfs,  we still have to check the file-system integrity running fsck, because the super-block free blocks field  is not properly updated after removing the file. And besides, we have an entire /proc/ physical file-system in there apart from that kcore file. So best to run a live GNU/Linux distro, mount the /dev/sda1 partition, remove the /proc/* directory recursively, and then let it boot up the usual way. I’ll do that next week, but  at least the mystery has finally solved.

A brief note a bit later on : it’s now fixed

Finally, I went to the place where this GNU/Linux box was located and I booted up a RIP CD/DVD live Linux distro, I mounted the slash partition, and I deleted all the /proc/* files and directories, including the /proc/kcore. And now, as you can imagine, the issue is completely fixed. Have a look at the df command output:

/dev/sda1              14G  7.2G  6.0G  55% /