Edit: Solved!
Unfortunately I’m not sure exactly what fixed it, because I was running btrfs commands like a madman.
Some combination of the following caused my 100GB labelled as UNREACHABLE to turn into UNUSED, which allowed that space to be written to as normal:
sudo btrfs balance start -v /
sudo btrfs filesystem defrag -v /
sudo btrfs filesystem defrag -v -r /
Also the tool btdu was incredibly helpful!
One of my linux boxes ran out of disk space, which surprised me, because it definitely didn’t have that much stuff on it. When I check with df it says I have used 212GB on my / path:
$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 227G 212G 5.2G 98% /
So, I tried to use du to see if maybe a runaway log file was the cause, but this says I have only used 101GB on my / path (this is also more in-line with how much space I expected to be used):
$ du -h | sort -h
...
101G /
Using those commands with sudo outputs the same sizes.
My filesystem is Btrfs, I’ve tried the suggestion to use btrfs balance start ... but this actually INCREASED my disk usage to 99% lol
So my question is… what on earth is using the remaining 111GB?? Why can I not see it in du?
I typically investigate with
ncduwhich gives very useful visualization like :--- /home/fabien/Prototypes/esphome/.esphome ---------------------------------------------------------------------------------------------------------------- /.. 3.1 GiB [######################] /platformio 218.1 MiB [# ] /build 28.0 KiB [ ] /idedata 8.0 KiB [ ] /storageand let’s you iterate. Here for example you’d go into
platformioand get another view, pressdto delete files or directories that aren’t needed anymore if it’s a stale project e.g.node_modules. Go back, etc.So yes, warmly recommended, both on desktop and remote servers. It’s way easier IMHO that
du -sh ./directorythencd, rinse and repeat. It’s also way WAY faster then GUI equivalents … because you navigate and take action, e.g. delete, with your keyboard.All that being said, if it’s about your filesystem rather than your files, it probably won’t help much. I don’t know enough about btrfs to help unfortunately.
ncdu
Oh this one is very cool! Unfortunately it also only shows the same 101GB being used:
ncdu 1.22 ~ Use the arrow keys to navigate, press ? for help --- / ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 93.1 GiB [###########################] /home 6.5 GiB [# ] /usr 790.4 MiB [ ] /var 173.0 MiB [ ] /boot 12.8 MiB [ ] /etc 1.7 MiB [ ] /root 1.3 MiB [ ] /run 44.0 KiB [ ] /tmp @ 4.0 KiB [ ] initrd.img.old @ 4.0 KiB [ ] initrd.img @ 4.0 KiB [ ] vmlinuz.old @ 4.0 KiB [ ] vmlinuz @ 4.0 KiB [ ] lib64 @ 4.0 KiB [ ] sbin @ 4.0 KiB [ ] lib @ 4.0 KiB [ ] bin . 0.0 B [ ] /proc 0.0 B [ ] /sys 0.0 B [ ] /dev 0.0 B [ ] /media e 0.0 B [ ] /srv e 0.0 B [ ] /opt e 0.0 B [ ] /mnt
btduis an excellent tool for finding out what’s taking up space in btrfsLegend! It found a second filesystem named “UNREACHABLE”:
It looks like an exact duplicate of my main filesystem “/@rootfs”, I’m guessing this is why my disk space filled up. Do you know how I’d go about removing the duplicate? (If it’s safe to do so)
There could be btrfs stuff I’m not aware of, but on a file system structure level, do you have a separate drive for booting and then another you added and mounted separately? Or did you install Linux over another install and changed partitions used? The reason I’m asking is you could have a whole drive of data under a folder and then later mount another partition or drive to that same folder. Linux will show you the mounted folder contents, but the original is not visible until you unmount your Mount point. The data is still there. So drive can be full, even though contents look smaller.
I can’t say its that for sure, but it has tripped people up before.
But could be btrfs cleanup needs looking at.
Just one drive, it was a clean Linux install with no funky stuff going on. I’ll have to look into Btrfs cleanup more, last time I did it the disk just filled up even more
Yeah, I haven’t dealt with that much since OpenSUSE has all the cleanup and maintenance scheduled.
Is your whole drive BTRFS even your /Home on same partition? The BTRFS subvolumes share a pool if setup as one, and filling up home subvolume also fills up your system subvolume, since its shared.
Strange, I’ve never seen that. Have you rebooted the system to make sure it has nothing to do with open files?
I did find one thread that seems related:
https://www.reddit.com/r/btrfs/comments/lip3dk/unreachable_data_on_btrfs_according_to_btdu/
BTRFS snapshots world be my first guess. ‘sudo btrfs subvolume list /‘
There is one listed:
ID 256 gen 137604 top level 5 path @rootfsLooks like it is just my filesystem though?
The bit of information you’re missing is that
duaggregates the size of all subfolders, so when you saydu /, you’re saying: “how much stuff is in / and everything under it?”If you’re sticking with
du, then you’ll need to traverse your folders, working downward until you find the culprit folder:$ du /* (Note which folder looks the biggest) $ du /home/* (If /home looks the biggest)… and so on.
The trouble with this method however is that
*won’t include folders with a.in front, which is often the culprit:.cache,.local/share, etc. For that, you can do:$ du /home/.*Which should do the job I think.
If you’ve got a GUI though, things get a lot easier 'cause you have access to GNOME Disk Usage Analyzer which will draw you a fancy tree graph of your filesystem state all the way down to the smallest folder. It’s pretty handy.
GUI disk space analyzers are absolutely amazing.
For those who prefer KDE and/or donut graphs, Filelight has you covered.
