Community Forum

Weavind
Weavind
Offline
Resolved
0 votes
My system (ClearOS V7.3.0) has started popping up with the message
Ooooops: Server is running low on available disk space - preventing further modifications via API until disk space issues are resolved.
.

My system summary shows the /dev/mapper/clearos-root as being 100% used. This is a 1Tb solid state drive. The biggest directory is my VAR at 165Gb, but the next biggest is the USR at only 2.3Gb. In total, i shouldn't be using more than 200Gb.


Filesystem Size Used Available Use % Mounted
/dev/mapper/clearos-root 923G 916G 6.8G 100% /
/dev/sda1 1.8T 368G 1.4T 22% /Backups
/dev/sdb1 497M 234M 264M 48% /boot
/dev/sdc1 1.8T 444G 1.3T 26% /B_One
/dev/sdd1 1.8T 438G 1.3T 26% /B_Two


I ran the command below, but this didn't highlight anything other than my flexshare folders.
du -axh /* | sort -hr | head -n20


I just can't seem to see where 916Gb is being used. Any ideas?
Wednesday, August 30 2017, 01:16 PM

Location [ View Larger Map ]

Share this post:
Responses (10)
  • Accepted Answer

    Tuesday, September 05 2017, 12:03 PM - #Permalink
    Resolved
    0 votes
    When you moved (copied?) your three folders to , did you delete the original folders before you did the "mount bind"? If you did not, the original folders are still taking up space and there is no way to directly get to them to recover the space. If the original files are still there, you need to unmounts the binds, delete the original files and then remount the binds.

    Don't worry that you've moved the flexshares and they are still being reported under /var/flexshare. That is where they logically belong in the file system, but physically they are on /dev/sda1.

    I need to learn the LVM stuff to work out what is happening with the device-mapper. I don't think it is very much. I think it is on /dev/sdb2 and contains two partitions, swap and the rest (including the original /var). As it is using the LVM you could have used the LVM manager (mdadm) to add one of your other disks to the volume to increase the volume of the apparent disk. I don't like this as it means you have no idea which disk any file is on (and presumably big files can get split between disks). Also I think it is a bad idea in your case as you will have no control over which files are on the SSD and which on the conventional disk so you could end up losing all the benefits of a high speed disk.
    The reply is currently minimized Show
  • Accepted Answer

    Weavind
    Weavind
    Offline
    Tuesday, September 05 2017, 09:24 AM - #Permalink
    Resolved
    0 votes
    Hi,

    I tried that with little success.

    If i run the df -h command, then I see that the /dev/mapper.clearos-root is using 781Gb of the drive.

    [root@server ~]# df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/mapper/clearos-root 923G 781G 143G 85% /
    devtmpfs 7.9G 0 7.9G 0% /dev
    tmpfs 7.9G 12K 7.9G 1% /dev/shm
    tmpfs 7.9G 219M 7.7G 3% /run
    tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
    /dev/sdb1 497M 234M 264M 48% /boot
    tmpfs 1.6G 0 1.6G 0% /run/user/991
    /dev/sda1 1.8T 517G 1.2T 30% /Backups
    /dev/sdc1 1.8T 451G 1.3T 26% /B_One
    /dev/sdd1 1.8T 444G 1.3T 26% /B_Two
    tmpfs 1.6G 0 1.6G 0% /run/user/0


    If i then run the du -h --max-depth=1 / it points me to the /var/flexshare folder.


    [root@server ~]# du -h --max-depth=1 /
    209M /boot
    12K /dev
    du: cannot access â/proc/9369/task/9369/fd/4â: No such file or directory
    du: cannot access â/proc/9369/task/9369/fdinfo/4â: No such file or directory
    du: cannot access â/proc/9369/fd/4â: No such file or directory
    du: cannot access â/proc/9369/fdinfo/4â: No such file or directory
    0 /proc
    219M /run
    0 /sys
    36M /etc
    52K /root
    4.0K /tmp
    554G /var
    2.4G /usr
    664K /home
    0 /media
    0 /mnt
    0 /opt
    0 /srv
    0 /store
    517G /Backups
    451G /B_One
    444G /B_Two
    2.0T /

    [root@server ~]# du -h --max-depth=1 /var
    6.0G /var/lib
    2.2G /var/log
    0 /var/adm
    215M /var/cache
    1.3M /var/db
    0 /var/empty
    0 /var/games
    0 /var/gopher
    0 /var/local
    0 /var/nis
    0 /var/opt
    0 /var/preserve
    7.3G /var/spool
    4.0K /var/tmp
    0 /var/yp
    0 /var/kerberos
    4.0K /var/state
    279M /var/clearos
    0 /var/crash
    537G /var/flexshare
    758M /var/www
    4.0K /var/ftp
    0 /var/samba
    554G /var

    [root@server ~]# du -h --max-depth=1 /var/flexshare/shares
    988M /var/flexshare/shares/ddb
    23G /var/flexshare/shares/mdd
    36G /var/flexshare/shares/photos
    372G /var/flexshare/shares/backups
    0 /var/flexshare/shares/backupdrive
    8.0K /var/flexshare/shares/website3.com
    8.0K /var/flexshare/shares/website2.co.za
    285M /var/flexshare/shares/website4.co.za
    395M /var/flexshare/shares/website.co.za
    79M /var/flexshare/shares/website1.co.za
    107G /var/flexshare/shares/development
    537G /var/flexshare/shares


    However i have moved the data for /var/flexshare/shares/photos, /var/flexshare/shares/development and /var/flexshare/shares/backups to the 2Tb drive mounted on /Backups with bind mounts to the /var/flexshare/shares folder. Have I done something wrong in the fstab file?


    [root@server ~]# cat /etc/fstab

    #
    # /etc/fstab
    # Created by anaconda on Fri Feb 24 08:38:45 2017
    #
    # Accessible filesystems, by reference, are maintained under '/dev/disk'
    # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
    #
    /dev/mapper/clearos-root / xfs defaults 0 0
    UUID=67c9a92c-8930-4833-9eaa-140ee5e5ecdb /boot xfs defaults 0 0
    /dev/mapper/clearos-swap swap swap defaults 0 0
    /dev/sda1 /Backups ext4 defaults 0 0
    /Backups/Backups /var/flexshare/shares/backups none defaults,bind 0 0
    /Backups/Development /var/flexshare/shares/development none defaults,bind 0 0
    /Backups/Photos /var/flexshare/shares/photos none defaults,bind 0 0
    /dev/sdc1 /B_One ext4 defaults 0 0
    /dev/sdd1 /B_Two ext4 defaults 0 0
    /var/www/html /var/flexshare/shares/website.co.za none defaults,bind 0 0
    /var/www/virtual/website4.co.za /var/flexshare/shares/website4.co.za none defaults,bind 0 0
    /var/www/virtual/website3.com /var/flexshare/shares/website3.com none defaults,bind 0 0
    /var/www/virtual/website2.co.za /var/flexshare/shares/website2.co.za none defaults,bind 0 0
    /var/www/virtual/website1.co.za /var/flexshare/shares/website1.co.za none defaults,bind 0 0
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, September 05 2017, 08:41 AM - #Permalink
    Resolved
    0 votes
    Please can you repeat the:
    du -h --max-depth=1 /
    making sure you include the trailing "/" or as Dave indicated, just do a "cd /" first. You can then work your way down the folder structure each time homing in on the folder with the highest figure.

    As a couple of quick wins, have a look in /var/log and below and delete any old stuff you don't want. Also in your flexshares, have a look in your flexshare location (/var/flexshare/shares if you have not changed it) and look at your recycle bin (.trash? but I'm working from memory).

    I'll have another look at your output at lunchtime.
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, September 05 2017, 07:52 AM - #Permalink
    Resolved
    0 votes
    You will want to change directory to / in order to run 'du -h --max-depth=1'. ie.

    cd /
    du -h --max-depth=1

    From there, it will tell you which folder is the big consumer. Enter into the folder that is the big consumer with 'cd /directory_name' and then repeat

    du -h --max-depth=1

    You'll get the hang of it.

    Another useful command for us to understand the situation is going to be:

    df -h
    The reply is currently minimized Show
  • Accepted Answer

    Weavind
    Weavind
    Offline
    Tuesday, September 05 2017, 05:44 AM - #Permalink
    Resolved
    0 votes
    Hi,

    Thanks for the offer - I would happily accept help on this as it has me stumped. I've output the commands below:

    [root@server ~]# fdisk -l

    Disk /dev/sda: 2000.4 GB, 2000398934016 bytes, 3907029168 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk label type: dos
    Disk identifier: 0x00044493

    Device Boot Start End Blocks Id System
    /dev/sda1 2048 3907028991 1953513472 83 Linux

    Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes, 1953525168 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk label type: dos
    Disk identifier: 0x000cc1f8

    Device Boot Start End Blocks Id System
    /dev/sdb1 * 2048 1026047 512000 83 Linux
    /dev/sdb2 1026048 1953523711 976248832 8e Linux LVM

    Disk /dev/mapper/clearos-root: 991.2 GB, 991172755456 bytes, 1935884288 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes


    Disk /dev/mapper/clearos-swap: 8455 MB, 8455716864 bytes, 16515072 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes


    Disk /dev/sdc: 2000.4 GB, 2000398934016 bytes, 3907029168 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk label type: dos
    Disk identifier: 0xaf68a7f3

    Device Boot Start End Blocks Id System
    /dev/sdc1 2048 3907029167 1953513560 83 Linux

    Disk /dev/sdd: 2000.4 GB, 2000398934016 bytes, 3907029168 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk label type: dos
    Disk identifier: 0x982c3424

    Device Boot Start End Blocks Id System
    /dev/sdd1 2048 3907029167 1953513560 83 Linux



    [root@server ~]# du -h --max-depth=1
    0 ./.pki
    0 ./.spamassassin
    12K ./.ssh
    0 ./ssh
    52K .



    [root@server ~]# blkid
    /dev/sda1: LABEL="Backup" UUID="51f820dd-9055-47c2-8354-833db87cee3c" TYPE="ext4"
    /dev/sdb1: UUID="67c9a92c-8930-4833-9eaa-140ee5e5ecdb" TYPE="xfs"
    /dev/sdb2: UUID="eZP5HM-BJtM-cAoG-bn5J-mD1r-Lc6i-eKry1e" TYPE="LVM2_member"
    /dev/mapper/clearos-root: UUID="b4bbaf55-bbdc-47e3-944a-0d6be6b240c5" TYPE="xfs"
    /dev/mapper/clearos-swap: UUID="fc7a7371-c4e9-40ab-b1ad-1ef1e8f678ae" TYPE="swap"
    /dev/sdc1: UUID="a6f06139-fbec-4240-9a8e-40f5c77ba5f5" TYPE="ext4"
    /dev/sdd1: UUID="8fee4bac-ab11-49bd-a37e-7f98e4b2cc83" TYPE="ext4"



    [root@server ~]# cat /etc/fstab

    #
    # /etc/fstab
    # Created by anaconda on Fri Feb 24 08:38:45 2017
    #
    # Accessible filesystems, by reference, are maintained under '/dev/disk'
    # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
    #
    /dev/mapper/clearos-root / xfs defaults 0 0
    UUID=67c9a92c-8930-4833-9eaa-140ee5e5ecdb /boot xfs defaults 0 0
    /dev/mapper/clearos-swap swap swap defaults 0 0
    /dev/sda1 /Backups ext4 defaults 0 0
    /Backups/Backups /var/flexshare/shares/backups none defaults,bind 0 0
    /Backups/Development /var/flexshare/shares/development none defaults,bind 0 0
    /Backups/Photos /var/flexshare/shares/photos none defaults,bind 0 0
    /dev/sdc1 /B_One ext4 defaults 0 0
    /dev/sdd1 /B_Two ext4 defaults 0 0
    /var/www/html /var/flexshare/shares/website.co.za none defaults,bind 0 0
    /var/www/virtual/website4.co.za /var/flexshare/shares/website4.co.za none defaults,bind 0 0
    /var/www/virtual/website3.com /var/flexshare/shares/website3.com none defaults,bind 0 0
    /var/www/virtual/website2.co.za /var/flexshare/shares/website2.co.za none defaults,bind 0 0
    /var/www/virtual/website1.co.za /var/flexshare/shares/website1.co.za none defaults,bind 0 0



    [root@server ~]# findmnt
    TARGET SOURCE FSTYPE OPTIONS
    / /dev/mapper/clearos-root
    xfs rw,relat
    ââ/sys sysfs sysfs rw,nosui
    â ââ/sys/kernel/security securityfs security rw,nosui
    â ââ/sys/fs/cgroup tmpfs tmpfs ro,nosui
    â â ââ/sys/fs/cgroup/systemd cgroup cgroup rw,nosui
    â â ââ/sys/fs/cgroup/devices cgroup cgroup rw,nosui
    â â ââ/sys/fs/cgroup/net_cls,net_prio cgroup cgroup rw,nosui
    â â ââ/sys/fs/cgroup/pids cgroup cgroup rw,nosui
    â â ââ/sys/fs/cgroup/cpuset cgroup cgroup rw,nosui
    â â ââ/sys/fs/cgroup/memory cgroup cgroup rw,nosui
    â â ââ/sys/fs/cgroup/freezer cgroup cgroup rw,nosui
    â â ââ/sys/fs/cgroup/blkio cgroup cgroup rw,nosui
    â â ââ/sys/fs/cgroup/perf_event cgroup cgroup rw,nosui
    â â ââ/sys/fs/cgroup/hugetlb cgroup cgroup rw,nosui
    â â ââ/sys/fs/cgroup/cpu,cpuacct cgroup cgroup rw,nosui
    â ââ/sys/fs/pstore pstore pstore rw,nosui
    â ââ/sys/kernel/debug debugfs debugfs rw,relat
    â ââ/sys/kernel/config configfs configfs rw,relat
    ââ/proc proc proc rw,nosui
    â ââ/proc/sys/fs/binfmt_misc systemd-1 autofs rw,relat
    â ââ/proc/sys/fs/binfmt_misc binfmt_misc binfmt_m rw,relat
    ââ/dev devtmpfs devtmpfs rw,nosui
    â ââ/dev/shm tmpfs tmpfs rw,nosui
    â ââ/dev/pts devpts devpts rw,nosui
    â ââ/dev/hugepages hugetlbfs hugetlbf rw,relat
    â ââ/dev/mqueue mqueue mqueue rw,relat
    ââ/run tmpfs tmpfs rw,nosui
    â ââ/run/user/991 tmpfs tmpfs rw,nosui
    â ââ/run/user/0 tmpfs tmpfs rw,nosui
    ââ/var/flexshare/shares/website1.co.za
    /dev/mapper/clearos-root[/var/www/virtual/website1.co.za]
    xfs rw,relat
    ââ/var/flexshare/shares/website2.co.za
    /dev/mapper/clearos-root[/var/www/virtual/website2.co.za]
    xfs rw,relat
    ââ/var/flexshare/shares/website3.com /dev/mapper/clearos-root[/var/www/virtual/website3.com]
    xfs rw,relat
    ââ/boot /dev/sdb1 xfs rw,relat
    ââ/var/flexshare/shares/website.co.za /dev/mapper/clearos-root[/var/www/html]
    xfs rw,relat
    ââ/var/flexshare/shares/website4.co.za /dev/mapper/clearos-root[/var/www/virtual/website4.co.za]
    xfs rw,relat
    ââ/Backups /dev/sda1 ext4 rw,relat
    ââ/B_One /dev/sdc1 ext4 rw,relat
    ââ/B_Two /dev/sdd1 ext4 rw,relat
    ââ/var/flexshare/shares/development /dev/sda1[/Development]
    ext4 rw,relat
    ââ/var/flexshare/shares/backups /dev/sda1[/Backups] ext4 rw,relat
    ââ/var/flexshare/shares/photos /dev/sda1[/Photos] ext4 rw,relat
    The reply is currently minimized Show
  • Accepted Answer

    Monday, September 04 2017, 08:37 PM - #Permalink
    Resolved
    0 votes
    Do you want any help with this? If do, please give the output to:
    fdisk -l
    du -h --max-depth=1 /
    blkid
    cat /etc/fstab
    findmnt (but you can edit out the bits including and below /sys, /proc, /dev and /run)
    The reply is currently minimized Show
  • Accepted Answer

    Weavind
    Weavind
    Offline
    Monday, September 04 2017, 06:22 AM - #Permalink
    Resolved
    0 votes
    Hi,

    Thanks for the responses. No, I have not completely resolved this. I moved my flexshares to an external drive and then used bind mounts for them. This brought my disc usage down from 100% to <90%. It's not ideal, but at least I can change/modify configuration without the Ooops message. The disc usage tool shows I'm only using <40Gb where the system report shows I'm using 780Gb. I think i may get a replacement solid state and do a fresh install. Then I can check the drive without hampering the live server.

    Thanks!
    The reply is currently minimized Show
  • Accepted Answer

    Sunday, September 03 2017, 06:27 PM - #Permalink
    Resolved
    0 votes
    How are you getting on with this? Have you resolved it?
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, August 30 2017, 07:35 PM - #Permalink
    Resolved
    0 votes
    You should use bind mounts to mount storage like /var/flexshare/shares to other devices. The advantage of using bindmounts over simply mounting the structure in place is that you can divide as desired.

    Here is a howto article on the subject:

    https://www.clearos.com/resources/documentation/clearos/content:en_us:kb_o_storage_manipulation_using_bindmounts
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, August 30 2017, 05:34 PM - #Permalink
    Resolved
    0 votes
    I can't really help troubleshoot at the moment as I'm miles away from home, and logical volumes are not my thing. Have a look in /etc/fstab and try to figure out how thing are mounted. If you are using bind mounts, be wary. If you copy data over to folder in preparation for it to be binf mounter, then once you do the bind mount, if you have not already deleted (or moved elsewhere) the original data, all that space is lost. It can only be recovered if you unmount the bind mount and delete the original data. Logical volume stuff is held elsewhere, but start with fstab.
    The reply is currently minimized Show
Your Reply