Developers Documentation

×

Warning

301 error for file:https://clearos.com/dokuwiki2/lib/exe/css.php?t=dokuwiki&tseed=82873f9c9a1f5784b951644363f20ef8

User Tools

Site Tools


Command line backup to USB drive

This howto will illustrate a simple way to backup files to an external USB drive. This is not intended to be a complete backup solution but is a framework for both quick and dirty backups and some backups and also a framework for any app that may use the RSYNC process with CronTab to perform scheduled backups to opportunistic USB devices. This method should only be used in conjunction with regular inspection and backup testing and validation.

This document is currently under testing and awaiting feedback. Please direct your feed back to Dave Loper. When complete this warning will be removed.

Prerequisites

You should have some level of comfort with vi and other command line tools before using this method or at the least, try it out in a test environment. Even veteran command-line gurus should also familiarize themselves with the Centralized User Data model for ClearOS since we will create our mount points under this model and this model accounts for backup stores.

Setup Backup Media

Decide whether you want to have your media for your backup readable by windows or not. The advantage to using FAT is that you can read the disk drive with a Windows Machine. The advantage to using EXT3 is that you can backup permissions. But you don't get both advantages.

Setup EXT3 partition

1) Partition USB or other removable drive in ClearOS or other linux distribution.

a) in Linux you would identify the drive using 'fdisk -l'
b) then using fdisk, create the partition. Use type **83** for the partition

2) Validate the drive partition in linux (fdisk -l)

fdisk -l

You should see something like this.

Disk /dev/sdc: 8011 MB, 8011120640 bytes
41 heads, 41 sectors/track, 9307 cylinders
Units = cylinders of 1681 * 512 = 860672 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               5        9308     7819328   83  Linux

3) Format the disk

mkfs.ext3 /dev/sdc1

4) Give the drive a label

mlabel -i /dev/sd*1 -s :: #### Replace the * with the drive letter in linux given for the drive (Step 2)
e2label /dev/sdc1 VOLUMENAME

5) Jump to step 5 below.

Setup FAT partition

1) Partition USB or other removable drive (ok to do in Windows if works)

a) in Linux you would identify the drive using 'fdisk -l'
b) then using fdisk, create the partition. Give it type **c** for the partition

2) Validate the drive partition in linux (fdisk -l)

fdisk -l

You should see something like this.

Disk /dev/sdc: 8011 MB, 8011120640 bytes
41 heads, 41 sectors/track, 9307 cylinders
Units = cylinders of 1681 * 512 = 860672 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               5        9308     7819328    c  W95 FAT32 (LBA)

3) Give the drive a label, can be done in Window (maybe)

mlabel -i /dev/sd*1 -s ::

#### Replace the * with the drive letter in linux given for the drive (Step 2)

You may not have the package that provides mlabel. If you get a message saying command not found, run 'yum install mtools'.

If you receive a warning 'Total number of sectors (###) not a multiple of sectors per track (##)! Add mtools_skip_check=1 to your .mtoolsrc file to skip this test, type the following:

echo "mtools_skip_check=1" > /root/.mtoolsrc

And try running the 'mlabel' command again.

4) If the label name is wrong give the volume a label in Linux and validate:

mlabel -i /dev/sd*1 ::VOLUMENAME #### Replace the * with the drive letter in linux given for the drive (Step 2)
#### Replace the VOLUMENAME with the name desired
mlabel -i /dev/sd*1 -s :: #### Replace the * with the drive letter in linux given for the drive (Step 2)

Setup mount point

5) Make the landing directory for the VOLUMENAME

mkdir /store/VOLUMENAME && touch /store/VOLUMENAME/volume.not.mounted

6) Update the /etc/fstab to reflect the new VOLUMENAME

nano /etc/fstab

#### If nano is not installed, install it (yum install nano or use vi if you are more comfortable with that)

a) The line added will look like this:

LABEL=VOLUMENAME           /store/VOLUMENAME          vfat    defaults,noauto 0 0

#### Use the Control+O (^O) to write the file. Use Control+X (^X) to exit. (in vi, use 'i' to enter insert mode for editing and to exit editing, use :wq to write and quit)

7) Mount the USB drive for testing

mount /store/VOLUMENAME

8) Validate the USB is mounted by making sure it is listed when running the 'mount' command:

root@gateway ~]# mount

/dev/md1 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/md0 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/dev/sdc1 on /store/VOLUMENAME type vfat (rw)
/dev/sdc1 on /store/VOLUMENAME type vfat (rw)    **** Look for this one
/dev/sdc1 on /store/VOLUMENAME type ext3 (rw)    **** or this one

9) Make the two folders and the validation file on the USB drive that is mounted

mkdir /store/VOLUMENAME/backup /store/backup1/backup/gateway
touch /store/VOLUMENAME/VOLUMENAME.do.not.delete.me

10) Validate that the file and directories are made using:

ls -laR /store/VOLUMENAME/*

11) Unmount the drive

umount /store/VOLUMENAME

12) All done, refer to the section about setting up the scripts and schedules.

Setup Backup Scripts

1) Copy the scripts from a system which is working well to the '/root' folder on the target server or make the scripts using your favorite editor

The files required are:

  • backupdaily.sh
  • backupweekly.sh
  • backup.includes

A copy of these files are included in the Appendix. Be sure to modify some of the static constants in the files.

2) Make sure that the script files are executable by running the following:

chmod +x /root/backupdaily.sh /root/backupweekly.sh

3) As needed, modify the 'backup.includes' to suit your needs.

4) Install mutt.

yum --enablerepo=clearos-core install mutt

5) Install the SMTP server module in Webconfig under ClearCenter » Modules.

6) Refresh you browser and navigate to Server > Mail > SMTP Server. Start the server and set it to automatically start.

Setup Backup Scheduler

1) The scheduler uses CRON to run jobs. You can use the template below to populate your initial crontab (a tablature of the scheduled jobs)

2) To edit the crontab run the following:

crontab -e

3) This will place you in the 'vi' (VIsual text editor)

a) 'vi' has to basic modes; command mode and insert mode. The starting mode is command mode
b) press 'i' to enter insert mode. You can tell that you are in insert mode by the '-- INSERT --' comment at the bottom of the screen
c) press  at any time to go into command mode

4) While in insert mode, type or paste the data from the template.

5) Once the data in listed on the screen, exit insert mode by pressing . To save the data

 type the following (it will appear at the bottom of the screen):

:wq

6) Press . The screen will breifly flash the message 'Installing crontab'

Crontab Template
0       2       *       *       2       /root/backupdaily.sh backup1 Monday
0       2       *       *       3       /root/backupdaily.sh backup1 Tuesday
0       2       *       *       4       /root/backupdaily.sh backup1 Wednesday
0       2       *       *       5       /root/backupdaily.sh backup1 Thursday
0       2       *       *       6       /root/backupweekly.sh backup1 
0       2       *       *       2       /root/backupdaily.sh backup2 Monday
0       2       *       *       3       /root/backupdaily.sh backup2 Tuesday
0       2       *       *       4       /root/backupdaily.sh backup2 Wednesday
0       2       *       *       5       /root/backupdaily.sh backup2 Thursday
0       2       *       *       6       /root/backupweekly.sh backup2

Info: Column meanings:

  • 1st: Minute
    • The first column is the minute column. Entries in this column range from 0-59 and represents the minute the command should be executed. If an asterisk '*' is present then EVERY minute is assumed. If you want something to execute every 5 minutes you'd use '*/5'. When specified as 0 the command will issue at the top of the hour. Cron honors ALL of the columns simultaneously so each column's condition must be met for it to run.
  • 2nd: Hour
    • As with the 1st column but represents the hour. Valid entries are 0-23.
  • 3rd: Day
    • As with the 1st column but represents the day of the month. Valid entries are 1-31.
  • 4th: Month
    • As with the 1st column but represents the mount. Valid entries are 1-12.
  • 5th: Day of week
    • As with the 1st column but represents the day of the week. Valid entries are 0-7 (both 0 AND 7 equal Sunday).
  • Last: What do I run
    • This is the command that is run and whitespace hereafter is passed at the command line. All commands are run as root unless otherwise directed.

For additional technical information please visit

Usage: In our usage we want to execute backups at 2 AM each morning. To do this we specify '0' for the minute and '2' for the hour. We place an asterisk '*' on the day of the month column and also on the month of the year column because we want to have the script run regardless of the day of the month or the month of the year. What is important is that these backups occur on specific days of the week and we specify those in the fifth column accordingly.

The paramaters we pass are those required by the backup scripts. The daily backup script requires that we specify which volume is the target (e.g. backup1 or backup2) and which day of the week we are going to use (i.e Monday through Friday). For the weekly backup we only need to specify the volumen name.

Please refer to the USB labelling document to match the volume name with the mount point in /etc/fstab.

Appendix

backupdaily.sh

#!/bin/bash

device=$1
#### replace this value 'gateway' with the pingable name of your server.
servername=gateway
#### replace this value 'helpdesk@example.com' with a valid email address.
notifyemail=helpdesk@example.com


if [ "$1" = "" ]; then
echo Syntax: /root/backupdaily [DEVICENAME] [BACKUPDAY]
echo EXAMPLE: /root/backupdaily backup1 Sunday
exit 0
fi

#### Mounting Backup Device
alreadymounted=`/bin/mount|grep $device`
if [ -n "$alreadymounted" ]; then
	echo Backup device already mounted
else
echo Attempting mount of Backup device
/bin/mount /store/$device
fi
if [ -f /store/$device/$device.do.not.delete.me ]; then
echo Backup device validated
else
echo Backup device invalid or not inserted correctly. Please contact the administrator
exit 0
fi

alreadymounted=`/bin/mount|grep $device`

#### Setup day of the Week ####

if [ -z $2 ]; then
echo "Backup using default day of Monday"
day=monday
else
day=$2
if [[ "$day" =~ "((Mon)|(Tues)|(Wednes)|(Thurs)|(Fri)|(Satur)|(Sun))day" ]]; then
	datevalid=1
else
	datevalid=0
fi

fi

#### Determine FAT or EXT4

vfatfs=`echo $alreadymounted|grep vfat`
extfs=`echo $alreadymounted|grep ext`

#### Backup
folder=/store/$device/backup/$servername
backupdir=$folder/$day
logfile="`date '+%d%B%Y'`".log
errorfile="`date '+%d%B%Y'`".error.log
backuplog=/tmp/$device.$logfile.log
errorlog=/tmp/$device.$errorfile.log
if [ ! -d $backupdir ]; then
mkdir $backupdir
fi

if [ ! -d $backupdir ]; then
      echo Backup failed, check layout of backup media
else
if [ "$datevalid" != "1" ]; then
echo Date invalid, check run script.
else
if [ -n "$extfs" ]; then
      	echo "EXT"
fi
if [ -n "$vfatfs" ]; then
	rm -rf $backupdir/*
	ls -laR / > $backupdir/directory.permissions.log
	echo Backup started. `date`
              /usr/bin/rsync -rtv --include-from=/root/backup.includes / $backupdir/ 2> $errorlog
	echo Backup complete. `date`
fi
fi
fi

#### Notify ####
touch $backuplog 
date > $backuplog
echo " " >> $backuplog
echo "`df -h|grep Filesystem`" >> $backuplog
echo "`df -h |grep /store/$device`" >> $backuplog
echo " " >> $backuplog

if [ -f $errorlog ]; then
echo "Errors:" >> $backuplog
echo "`cat $errorlog`" >> $backuplog
echo " " >> $backuplog
fi

if [ -d $backupdir/var ]; then
status="Backup SUCCESSFUL!"
echo $status >> $backuplog
echo " " >> $backuplog
else
status="Backup FAILED!"
echo $status >> $backuplog
echo " " >> $backuplog
fi

/bin/cp $backuplog $backupdir/$logfile

host=`hostname`

if [ -f $backupdir/$logfile ]; then
echo "$status $host - $device" | /usr/bin/mutt $notifyemail -a /$backupdir/$logfile -s "$status"
else
echo "Backup script failed. $host - $device" | /usr/bin/mutt $notifyemail -a $backuplog -s "Backup script failed, contact support"
fi

echo "Files backed up listed below: " >> $backupdir/$logfile
echo " " >> $backupdir/$logfile
ls -laR $backupdir >> $backupdir/$logfile

sleep 5
rm $backuplog
rm $errorlog

sync
sleep 2

#### Unmounting USB device ####
/bin/umount /store/$device
sleep 4
alreadymounted=`/bin/mount|grep $device`
if [ -n "$alreadymounted" ]; then
      echo Backup device failed to dismount $device, attempting dismount again
      /bin/umount /store/$device
else
      echo Device $device: unmounted successfully.
      exit 0
fi
 
sleep 4
alreadymounted=`/bin/mount|grep $device`
if [ -n "$alreadymounted" ]; then
      echo Unable to dismount $device. Contact your administrator
      echo `lsof /store/$device`
else
      echo Device '$device': unmounted successfully.
      exit 0
fi

backupweekly.sh

device=$1
#### replace this value 'gateway' with the pingable name of your server.
servername=gateway
#### replace this value 'helpdesk@example.com' with a valid email address.
notifyemail=helpdesk@example.com

if [ "$1" = "" ]; then
      echo Syntax: /root/backupweekly [DEVICENAME] 
      echo EXAMPLE: /root/backupweekly backup1 
      exit 0
fi

#### Mounting Backup Device
alreadymounted=`/bin/mount|grep $device`
if [ -n "$alreadymounted" ]; then
echo Backup device already mounted
else
echo Attempting mount of Backup device
/bin/mount /store/$device
fi
if [ -f /store/$device/$device.do.not.delete.me ]; then
echo Backup device validated
else
echo Backup device invalid or not inserted correctly. Please contact the administrator
exit 0
fi

alreadymounted=`/bin/mount|grep $device`

day=`date '+%d%B%Y'`
echo $day
datevalid=1

#### Determine FAT or EXT4

vfatfs=`echo $alreadymounted|grep vfat`
extfs=`echo $alreadymounted|grep ext`

#### Backup
folder=/store/$device/backup/$servername
backupdir=$folder/$day
logfile="`date '+%d%B%Y'`".log
errorfile="`date '+%d%B%Y'`".error.log
backuplog=/tmp/$device.$logfile.log
errorlog=/tmp/$device.$errorfile.log

if [ ! -d $backupdir ]; then
mkdir $backupdir
fi

if [ ! -d $backupdir ]; then
      echo Backup failed, check layout of backup media
else
if [ "$datevalid" != "1" ]; then
echo Date invalid, check run script.
else
if [ -n "$extfs" ]; then
      	echo "EXT"
fi
if [ -n "$vfatfs" ]; then
	rm -rf $backupdir/*
	ls -laR / > $backupdir/directory.permissions.log
	echo Backup started. `date`
	/usr/bin/rsync -rtv --include-from=/root/backup.includes / $backupdir/ 2> $errorlog
	echo Backup complete. `date`
fi
fi
fi

sync
sleep 2

#### Notify ####
touch $backuplog
date > $backuplog
echo " " >> $backuplog
echo "`df -h|grep Filesystem`" >> $backuplog
echo "`df -h |grep /store/$device`" >> $backuplog
echo " " >> $backuplog

if [ -f $errorlog ]; then
    echo "Errors:" >> $backuplog
    echo "`cat $errorlog`" >> $backuplog
    echo " " >> $backuplog
fi

if [ -d $backupdir/var ]; then
    status="Backup SUCCESSFUL!"
    echo $status >> $backuplog
    echo " " >> $backuplog
else
    status="Backup FAILED!"
    echo $status >> $backuplog
    echo " " >> $backuplog
fi

/bin/cp $backuplog $backupdir/$logfile

host=`hostname`

if [ -f $backupdir/$logfile ]; then
    echo "$status $host - $device" | /usr/bin/mutt $notifyemail -a /$backupdir/$logfile -s "$status"
else
    echo "Backup script failed. $host - $device" | /usr/bin/mutt $notifyemail -a $backuplog -s "Backup script failed, contact support"
fi

echo "Files backed up listed below: " >> $backupdir/$logfile
echo " " >> $backupdir/$logfile
ls -laR $backupdir >> $backupdir/$logfile

sleep 5
rm $backuplog
rm $errorlog

#### Unmounting USB device ####
/bin/umount /store/$device
sleep 4
alreadymounted=`/bin/mount|grep $device`
if [ -n "$alreadymounted" ]; then
      echo Backup device failed to dismount $device, attempting dismount again
/bin/umount /store/$device
else
echo Device $device: unmounted successfully.
exit 0
fi

sleep 4
alreadymounted=`/bin/mount|grep $device`
if [ -n "$alreadymounted" ]; then
      echo Unable to dismount $device. Contact your administrator 
echo `lsof /store/$device`
else
echo Device '$device': unmounted successfully.
exit 0
fi

backup.includes

+ /var/flexshare/shares/**
+ /var/flexshare/shares/
+ /var/flexshare/
+ /var/
+ /home/**
+ /home/
content/en_us/kb_o_local_backup_of_data_to_usb_-_command_line_style.txt · Last modified: 2015/03/01 15:15 (external edit)

https://clearos.com/dokuwiki2/lib/exe/indexer.php?id=content%3Aen_us%3Akb_o_local_backup_of_data_to_usb_-_command_line_style&1710833788