Community Forum

Resolved
0 votes
My system drive / is reported 95 % full but I can't find the data? I use the storage manager app and my /store drive is a 3,7 TB raid 0 set (2disks of 2 TB)
This is my fstab file. Does anyone have an idea where the storage is gone. One website is running piwigo (photo gallery) Is apache using a cache somewhere?


# /etc/fstab
# Created by anaconda on Sat Nov 28 18:37:00 2015
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=2618a046-270f-4ea6-ac5d-6d0c321f58bb / xfs defaults 0 0
UUID=73d22a26-75ba-4449-bd43-a25152e87719 /boot xfs defaults 0 0
UUID=121d6a4b-6890-4b76-980b-8af48b08d97c /store xfs defaults 0 0
UUID=1d9675f7-070d-477a-b880-3a7c5f109e13 swap swap defaults 0 0
# Storage engine - start
/store/flexshare/shares /var/flexshare/shares none bind,rw 0 0
/store/mysql/database /var/lib/mysql none bind,rw 0 0
/store/system_database/database /var/lib/system-mysql none bind,rw 0 0
/store/users/home /home none bind,rw 0 0
# Storage engine - end
/var/www/virtual/soap.vroemen.eu /var/flexshare/shares/soap.vroemen.eu none defaults,bind 0 0
/var/www/virtual/sint.vroemen.eu /var/flexshare/shares/sint.vroemen.eu none defaults,bind 0 0
/var/www/virtual/media.vroemen.eu /var/flexshare/shares/media.vroemen.eu none defaults,bind 0 0
/var/www/virtual/geschiedenis.scowo.nl /var/flexshare/shares/geschiedenis.scowo.nl none defaults,bind 0 0
/var/www/virtual/securitybeheer.vroemen.eu /var/flexshare/shares/securitybeheer.vroemen.eu none defaults,bind 0 0
/var/www/virtual/2010.scowo.nl /var/flexshare/shares/2010.scowo.nl none defaults,bind 0 0
/var/www/virtual/2000.scowo.nl /var/flexshare/shares/2000.scowo.nl none defaults,bind 0 0
/var/www/html /var/flexshare/shares/frogserver.vroemen.eu none defaults,bind 0 0
/var/www/virtual/soap2.vroemen.eu /var/flexshare/shares/soap2.vroemen.eu none defaults,bind 0 0
Sunday, May 14 2017, 08:53 AM
Share this post:
Responses (23)
  • Accepted Answer

    Tuesday, May 23 2017, 06:26 PM - #Permalink
    Resolved
    0 votes
    I've just been playing round with the webconfig and unfortunately, reading the docs, I don't want to particularly enable storage manager on a running system. You should be able to put your fstab line just below the storage section in fstab. "mount -a" will then mount it.

    You can probably hack something to make it go in the webconfig as well but I am a little unsure of how to do it. I can see a file to copy as a template but I am unsure exactly how to edit it.
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, May 23 2017, 05:56 PM - #Permalink
    Resolved
    0 votes
    @Lex My personal suggestion (as it seems you are not in production yet) would be to drop the storage manager - this is not because it is a bad app, but because it adds an extra layer of complexity. As your 2Tb is already in a RAID you can treat it as a partition and mount your various partitions as you please.

    Just my 2 cents
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, May 23 2017, 05:34 PM - #Permalink
    Resolved
    0 votes
    Sorry I haven't got any possibility in the webgui to ad a mapping so I thought that I have to do it by adding it to fstab but that's not working so there might be something wrong.For the moment I have got it back as before maybe first a test server to test.
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, May 23 2017, 04:13 PM - #Permalink
    Resolved
    0 votes
    So you are saying you had to manually add the entry in fstab rather than use the webconfig? If so, did you activate the entry with a "mount -a"? What, then, is the output of
    findmnt | grep  "\["
    ls -l /store/www/ | grep '^d'
    ls -l /var/www/ | grep '^d'
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, May 23 2017, 12:45 PM - #Permalink
    Resolved
    0 votes
    Nick I tried as you discriped but for some sort of reason changing /etc/fstab to make the new bind/ woudn't work
    I added the line under storage engine
    /store/www /var/www bind none 0 0
    but it didn't work after the change I did "storage" and after there where lines added to /etc/fstab
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, May 18 2017, 12:41 PM - #Permalink
    Resolved
    0 votes
    I think the bind mounts are OK. Everything under /var/www is on your SSD, but bound to flexshares so they look like they are on /store but are not.

    Please read the documentation on the Storage Manager and the Technical Guide at the bottom of it to get an idea what it is doing

    Then I'd suggest you do the following:
    Stop the web server
    Make a backup of everything in /var/www in one of your home drives. Then the backup is on the RAID.
    Create a /store/www
    Move (not copy) everything under /var/www to under /store/www
    In the Storage Manager map /store/www into /var/www
    Check you can see your webserver files under /var/www again.
    Restart the web server.
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, May 18 2017, 09:13 AM - #Permalink
    Resolved
    0 votes
    Well everything is working back to as before. /home /var/flexshare/shares are both on the raid set MD127 so thats fine the only point is that the websites are in /var/www/virtual/ on sda2 which is a SSD with limited space. But for now I fine with it since the migration from gallery3 to piwigo went well and I good dump the old site (14gb).
    To me it looks like that these lines:
    /var/www/virtual/soap.vroemen.eu /var/flexshare/shares/soap.vroemen.eu none defaults,bind 0 0
    /var/www/virtual/sint.vroemen.eu /var/flexshare/shares/sint.vroemen.eu none defaults,bind 0 0
    /var/www/virtual/media.vroemen.eu /var/flexshare/shares/media.vroemen.eu none defaults,bind 0 0
    /var/www/virtual/geschiedenis.scowo.nl /var/flexshare/shares/geschiedenis.scowo.nl none defaults,bind 0 0
    /var/www/virtual/securitybeheer.vroemen.eu /var/flexshare/shares/securitybeheer.vroemen.eu none defaults,bind 0 0
    /var/www/virtual/2010.scowo.nl /var/flexshare/shares/2010.scowo.nl none defaults,bind 0 0
    /var/www/virtual/2000.scowo.nl /var/flexshare/shares/2000.scowo.nl none defaults,bind 0 0
    /var/www/html /var/flexshare/shares/frogserver.vroemen.eu none defaults,bind 0 0
    /var/www/virtual/soap2.vroemen.eu /var/flexshare/shares/soap2.vroemen.eu none defaults,bind 0 0
    Just aint working,
    Would removing these and adding a line in sorage like
    /store/virtual/ /var/www/virtual/ none bind,rw 0 0
    in the storage engine part solve the problem obvious before the change I have to move the contents of /var/www/virtual/
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, May 18 2017, 08:09 AM - #Permalink
    Resolved
    0 votes
    Yes it is tortuous, but it looks like the home files are physically in /store/users/home then mapped into /home. I'd expect /home and /store to be on the same device. The way I'd find the UUID/device conversion is by doing "ls -l /dev/disk/by-uuid".
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, May 18 2017, 05:57 AM - #Permalink
    Resolved
    0 votes
    Nick I am likewise struggling, but

    as per fstab

    /store/flexshare/shares /var/flexshare/shares none bind,rw 0 0
    - my understanding is that everything under /var/flexshare/shares is actually located in /store/flexshares/shares, which in turn is physically located on ???. Looking at the output of df -h I cannot see where /store is mounted - the large volume seems to hold /home

    Filesystem Size Used Avail Use% Mounted on
    /dev/sda2 54G 50G 4.2G 93% /
    devtmpfs 2.9G 4.0K 2.9G 1% /dev
    tmpfs 2.9G 0 2.9G 0% /dev/shm
    tmpfs 2.9G 17M 2.9G 1% /run
    tmpfs 2.9G 0 2.9G 0% /sys/fs/cgroup
    /dev/md127 3.4T 1.1T 2.4T 31% /home


    In /etc/fstab /store is it mounted on UUID =121d6a4b-6890-4b76-980b-8af48b08d97c and the output of
    df /var/flexshare/shares/
    Filesystem 1K-blocks Used Available Use% Mounted on
    /dev/md127 3636921248 1108900664 2528020584 31% /var/flexshare/shares
    shows that /var/flexshare/shares is physically on

    /dev/md127 which according to df -h is where /home is mounted!!


    @Lex can you post the output of
     blkid
    and
    mount
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, May 17 2017, 05:48 PM - #Permalink
    Resolved
    0 votes
    I'm struggling to get my head round this one and I'm trying to work it through now I'm home.

    It looks like /store/flexshare/shares is mounted into /var/flexshare/shares and /store is your RAID array. That puts /var/flexshare/shares on your array.
    Then /var/www/virtual/soap.vroemen.eu is mounted into /var/flexshare/shares/soap.vroemen.eu that leaves your web server files in /var/www/virtual/soap.vroemen.eu but they appear to be in /var/flexshare/shares/soap.vroemen.eu. This means its files are on /dev/sda2.

    Similarly /var/www/html is mounted into /var/flexshare/shares/frogserver.vroemen.eu so its files are in /var/www/html which is on /dev/sda2 as well.

    If this is correct, and please someone confirm it (Duncan?), then the best strategy would be to move /var/www in its entirety into /store - and remember to delete the files from /var/www once they are also under /store before you do the bind mount. If you are nervous, and probably for safety anyway, make a complete copy of /var/www somewhere else before you delete anything.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, May 17 2017, 10:18 AM - #Permalink
    Resolved
    0 votes
    Note, firstly I only said there *may* be a bug.

    Trying to read this again it may not be a bug if your web sites are really in /var/www/virtual as these have never been moved to /store. Have a look in the Storage manager App at the docs, and try moving /var/www/virtual into /store somewhere.

    From the earlier posts we may have been interpreting your output incorrectly as /var is not all on your /store. It only (largely) appears to be because of the mount binds.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, May 17 2017, 09:56 AM - #Permalink
    Resolved
    0 votes
    Oke done that but now the big question how do I move the directories from one disk to the other without breaking my web server etc..?
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, May 17 2017, 09:22 AM - #Permalink
    Resolved
    0 votes
    Looking at your original post, if the fstab is entirely produced by the system, then I *think* there may be a bug. The bug is that the storage manager entries need to go at the end of the file or the mappings to /var/www/virtual take precedence.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, May 17 2017, 09:09 AM - #Permalink
    Resolved
    0 votes
    [root@frogserver ~]# df /var/www/virtual
    Filesystem 1K-blocks Used Available Use% Mounted on
    /dev/sda2 56141024 51837632 4303392 93% /
    [root@frogserver ~]#
    [root@frogserver ~]# df /var/flexshare/shares/
    Filesystem 1K-blocks Used Available Use% Mounted on
    /dev/md127 3636921248 1108900664 2528020584 31% /var/flexshare/shares
    [root@frogserver ~]# df /var/flexshare/shares/soap2.vroemen.eu/
    Filesystem 1K-blocks Used Available Use% Mounted on
    /dev/sda2 56141024 51837720 4303304 93% /var/flexshare/shares/soap2.vroemen.eu
    [root@frogserver ~]#


    It almost looks like that the rules in fstab are working the other way around!
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, May 17 2017, 08:48 AM - #Permalink
    Resolved
    0 votes
    what is the output of
    df /var/www/virtual 


    If that comes back that it is mounted on / then that is the issue
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, May 17 2017, 08:34 AM - #Permalink
    Resolved
    0 votes
    There is something strange I'll tell you what I've done. I moved the directories in /var/www/virtual to /store/tmp/virtual and that solved the problem of an allmost full disk. And with moving it back the problem returned. (and my webserver is screwed up, when I change something in the gui I get an oops). So it seems to mee that there is something going wrong with the storage app and the way the webserver is using a bind.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, May 17 2017, 06:26 AM - #Permalink
    Resolved
    0 votes
    Yes, I would follow Nick's suggestion because df is telling you that the you have 50Gb of data physically located on the /dev/sda2 partition
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, May 16 2017, 06:12 PM - #Permalink
    Resolved
    0 votes
    I think you may need to try what I said and check for lost space.
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, May 16 2017, 05:26 PM - #Permalink
    Resolved
    0 votes
    df -h shows this
    Filesystem Size Used Avail Use% Mounted on
    /dev/sda2 54G 50G 4.2G 93% /
    devtmpfs 2.9G 4.0K 2.9G 1% /dev
    tmpfs 2.9G 0 2.9G 0% /dev/shm
    tmpfs 2.9G 17M 2.9G 1% /run
    tmpfs 2.9G 0 2.9G 0% /sys/fs/cgroup
    /dev/md127 3.4T 1.1T 2.4T 31% /home
    /dev/sda1 473M 208M 266M 44% /boot
    tmpfs 580M 0 580M 0% /run/user/994
    tmpfs 580M 0 580M 0% /run/user/301
    tmpfs 580M 0 580M 0% /run/user/0
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, May 16 2017, 12:24 PM - #Permalink
    Resolved
    0 votes
    What does
    df -h
    show?
    The reply is currently minimized Show
  • Accepted Answer

    Sunday, May 14 2017, 11:45 AM - #Permalink
    Resolved
    0 votes
    I wouldn't worry about /var for the moment. The main constituent of /var is /flexshare which is on your RAID and is much too big to be on your SSD. As you say there is nothing big which stands out. The largest thing left on the ssd is /usr and that is only 5.9GB

    There is one big gotcha to watch out for when setting up /store. If you set it up, for example for /home, by copying everything over from /home to your new /store/users/home on /store, all the original /home files remain in place and once you bind /store/users/home into /home, this effectively becomes dead space.

    With your users off the system, you could try to unmount /home with the umount command. Check it is unmounted by running the "mount" command on its own (There may be a better command to show this run it before and after to compare). If it is not unmounted try to unmount /store/users/home instead. I can't remember which you have to unmount. When it is unmounted have another look to see what is in /home. You should be able to delete it all. Remount the storage location with a "mount -a"

    For mysql and system-mysql remember to stop the daemons before unmounting.

    [edit]
    See this for ideas on listing bind mounts like everything under /store.
    [/edit]
    The reply is currently minimized Show
  • Accepted Answer

    Sunday, May 14 2017, 09:42 AM - #Permalink
    Resolved
    0 votes
    Hi Nick done that but this is the output

    du: cannot access ‘./proc/23850/task/23850/fd/4’: No such file or directory
    du: cannot access ‘./proc/23850/task/23850/fdinfo/4’: No such file or directory
    du: cannot access ‘./proc/23850/fd/4’: No such file or directory
    du: cannot access ‘./proc/23850/fdinfo/4’: No such file or directory
    2.2T .
    1.1T ./var
    1.1T ./store
    30G ./home
    5.9G ./usr
    184M ./boot
    36M ./root
    34M ./etc
    8.7M ./run
    200K ./.readahead
    36K ./tmp
    4.0K ./dev
    0 ./sys
    0 ./srv
    0 ./sbin
    0 ./proc
    0 ./opt
    0 ./mnt
    0 ./media
    0 ./lib64
    0 ./lib
    0 ./bin
    0 ./.autorelabel

    So seeing that with the fstab in mind you would think it would be /var which is big.

    But /var is this;
    1.1T ./flexshare
    1.1T .
    41G ./www
    1.2G ./lib
    79M ./log
    40M ./clearos
    9.6M ./cache
    1.1M ./db
    156K ./spool
    4.0K ./.updated
    4.0K ./tmp
    4.0K ./state
    0 ./yp
    0 ./var
    0 ./samba
    0 ./run
    0 ./preserve
    0 ./opt
    0 ./nis
    0 ./mail
    0 ./lock
    0 ./local
    0 ./kerberos
    0 ./gopher
    0 ./games
    0 ./empty
    0 ./crash
    0 ./adm

    I am lost!
    The reply is currently minimized Show
  • Accepted Answer

    Sunday, May 14 2017, 09:29 AM - #Permalink
    Resolved
    0 votes
    Start at "/" then work your way down with:
    du -h --max-depth=1
    Each time change to the biggest directory with a "cd ......" and repeat the command.
    The reply is currently minimized Show
Your Reply