[PLUG] Rsync again

[John Jason Jordan]

Is there a way to stop an rsync command so that it gets to the end of
the file it is writing before it stops, less you end up with a
partially transferred file?

[John Jason Jordan]

On Sat, 6 Oct 2018 10:17:05 -0700
John Jason Jordan <***@gmx.com> dijo:

I need some advice with respect to the following command, which makes
my Synology NAS device a mirror of my external USB storage enclosure:

rsync -rptog --progress --stats --delete
--exclude-from=/media/jjj/Movies/rsync_exclusions /media/jjj/Movies/ /media/jjj/Synology

The first problem is that sometimes I get an error message that it
can't find /media/jjj/Movies. I found that a simple 'cd ..' followed by
'cd /media/jjj/Movies' solves the problem. I plan to convert the rsync
command above to a simple script, adding the cd commands in front of
the rsync command.

But there is a second, even worse problem: It sometimes can't find the
Synology NAS device. And if it can't find the device it dumps
everything into the local hard drive where /media/jjj/Synology is
(supposed to be) mounted. The hard drive has a couple hundred GB of free
space, but since the external USB device has over 7TB of data on it the
local disk is quickly filled up, and then the rsync command fails with
'the device is full.'

The Synology is mounted with this line in fstab:

192.168.1.115:/volume1/Synology /media/jjj/Synology nfs
auto,user 0 0

I need to add another command before the rsync command that checks to
be sure the Synology is mounted where it's supposed to be and mount it
there if it is not already mounted.

And finally, I need to add something after the rsync command that
checks to be sure the command executed as it is supposed to and give me
a popup that it did, or that it did not. I have gxmessage installed and
I could probably figure out how to make it pop up the messages, but I
don't know how to check that the rsync command executed properly. Could
I just check the two filesystems to see if they are equal in all
respects?

Suggestions?

[Tomas K]

The answer is - Do not start rsync until your NAS is mounted and
working.

Simple if statement in your backup script should take care of it.
You can do it number of ways - here is one - checking that there is any
content in the directory before starting rsync (in bash):

#!/bin/bash
backupTargetDir=/media/jjj/Synology
cd $backupTargetDir
# wait 30 seconds for the directory to mount
sleep 30
filesInDir=$(ls | wc -l)
if (( $filesInDir > 0 )); then
echo "INFO: Backing up files to $backupTargetDir"
rsync ....
else
echo "WARNING: No files in $backupTargetDir"
fi

Another way would be to use mount to verify that your NAS is mounted
....

Tomas

Post by John Jason Jordan
On Sat, 6 Oct 2018 10:17:05 -0700
I need some advice with respect to the following command, which makes
rsync -rptog --progress --stats --delete
--exclude-from=/media/jjj/Movies/rsync_exclusions
/media/jjj/Movies/ /media/jjj/Synology
The first problem is that sometimes I get an error message that it
can't find /media/jjj/Movies. I found that a simple 'cd ..' followed by
'cd /media/jjj/Movies' solves the problem. I plan to convert the rsync
command above to a simple script, adding the cd commands in front of
the rsync command.
But there is a second, even worse problem: It sometimes can't find the
Synology NAS device. And if it can't find the device it dumps
everything into the local hard drive where /media/jjj/Synology is
(supposed to be) mounted. The hard drive has a couple hundred GB of free
space, but since the external USB device has over 7TB of data on it the
local disk is quickly filled up, and then the rsync command fails with
'the device is full.'
192.168.1.115:/volume1/Synology /media/jjj/Synology nfs
auto,user 0 0
I need to add another command before the rsync command that checks to
be sure the Synology is mounted where it's supposed to be and mount it
there if it is not already mounted. 
And finally, I need to add something after the rsync command that
checks to be sure the command executed as it is supposed to and give me
a popup that it did, or that it did not. I have gxmessage installed and
I could probably figure out how to make it pop up the messages, but I
don't know how to check that the rsync command executed properly. Could
I just check the two filesystems to see if they are equal in all
respects?
Suggestions?
_______________________________________________
PLUG mailing list
http://lists.pdxlinux.org/mailman/listinfo/plug

[John Jason Jordan]

On Sat, 06 Oct 2018 13:08:21 -0700

Post by Tomas K
Simple if statement in your backup script should take care of it.
You can do it number of ways - here is one - checking that there is any
#!/bin/bash
backupTargetDir=/media/jjj/Synology
cd $backupTargetDir
# wait 30 seconds for the directory to mount
sleep 30
filesInDir=$(ls | wc -l)
if (( $filesInDir > 0 )); then
echo "INFO: Backing up files to $backupTargetDir"
rsync ....
else
echo "WARNING: No files in $backupTargetDir"
fi

¡Me gusta!

Here is how I finalized it:

#!/bin/bash
cd ..
cd /media/jjj/Movies
backupTargetDir=/media/jjj/Synology
cd $backupTargetDir
# wait 30 seconds for the directory to mount
sleep 30
filesInDir=$(ls | wc -l)
if (( $filesInDir > 0 )); then
echo "INFO: Backing up files to $backupTargetDir"
rsync -rptog --progress --stats --delete
--exclude-from=/media/jjj/Movies/rsync_exclusions /media/jjj/Movies/ /media/jjj/Synology
else echo "WARNING: No files in $backupTargetDir"
fi

I could probably add another if statement to be sure
that /media/JJJ/Movies is mounted, but that's my work directory, so
it's highly unlikely that it would not be mounted.

I tested it and it works fine! Thanks for the help!

[Larry Brigman]

I would add checking of the return code (exit code) for the 'cd' commands.
As it is now those could fail but something latter would have to detect the
failure.

Post by John Jason Jordan
On Sat, 06 Oct 2018 13:08:21 -0700

Post by Tomas K
Simple if statement in your backup script should take care of it.
You can do it number of ways - here is one - checking that there is any
#!/bin/bash
backupTargetDir=/media/jjj/Synology
cd $backupTargetDir
# wait 30 seconds for the directory to mount
sleep 30
filesInDir=$(ls | wc -l)
if (( $filesInDir > 0 )); then
echo "INFO: Backing up files to $backupTargetDir"
rsync ....
else
echo "WARNING: No files in $backupTargetDir"
fi

¡Me gusta!
#!/bin/bash
cd ..
cd /media/jjj/Movies
backupTargetDir=/media/jjj/Synology
cd $backupTargetDir
# wait 30 seconds for the directory to mount
sleep 30
filesInDir=$(ls | wc -l)
if (( $filesInDir > 0 )); then
echo "INFO: Backing up files to $backupTargetDir"
rsync -rptog --progress --stats --delete
--exclude-from=/media/jjj/Movies/rsync_exclusions /media/jjj/Movies/ /media/jjj/Synology
else echo "WARNING: No files in $backupTargetDir"
fi
I could probably add another if statement to be sure
that /media/JJJ/Movies is mounted, but that's my work directory, so
it's highly unlikely that it would not be mounted.
I tested it and it works fine! Thanks for the help!
_______________________________________________
PLUG mailing list
http://lists.pdxlinux.org/mailman/listinfo/plug

[Erik Lane]

That would be nice to be able to stop it in that way. I'm not a Linux guru,
so I'd also like to hear if there's some way to do that.

However, I just wanted to say that I always use -P as one of the options,
and this will resume partial downloads. So if you have to stop it and then
resume later, it will just pick up where it left off.

Not quite what you were asking for, but another way to solve the same
problem. (At least as far as I understood your question.)

Thanks,
Erik

Post by John Jason Jordan
Is there a way to stop an rsync command so that it gets to the end of
the file it is writing before it stops, less you end up with a
partially transferred file?
_______________________________________________
PLUG mailing list
http://lists.pdxlinux.org/mailman/listinfo/plug

[John Jason Jordan]

On Sat, 6 Oct 2018 12:23:04 -0700

Post by Erik Lane
That would be nice to be able to stop it in that way. I'm not a Linux
guru, so I'd also like to hear if there's some way to do that.
However, I just wanted to say that I always use -P as one of the
options, and this will resume partial downloads. So if you have to
stop it and then resume later, it will just pick up where it left off.
Not quite what you were asking for, but another way to solve the same
problem. (At least as far as I understood your question.)

Well, the problem resolved itself, at least in this case. The reason I
wanted to stop it was because I was watching and it was backing up
files that I was pretty sure had already been backed up, and the only
reason it might be doing that was if the backup device was not mounted.
As it turns out my guess was right, 15 minutes later rsync stopped
all by itself when it ran out of space on the mount point drive.

And this is not the first time I have copied things to a drive that I
thought was mounted, but was not. It's a perennial problem, and it's
high time I figured out a way to keep it from happening.

[Roderick Anderson]

Post by John Jason Jordan
On Sat, 6 Oct 2018 12:23:04 -0700

Post by Erik Lane
That would be nice to be able to stop it in that way. I'm not a Linux
guru, so I'd also like to hear if there's some way to do that.
However, I just wanted to say that I always use -P as one of the
options, and this will resume partial downloads. So if you have to
stop it and then resume later, it will just pick up where it left off.
Not quite what you were asking for, but another way to solve the same
problem. (At least as far as I understood your question.)

Well, the problem resolved itself, at least in this case. The reason I
wanted to stop it was because I was watching and it was backing up
files that I was pretty sure had already been backed up, and the only
reason it might be doing that was if the backup device was not mounted.
As it turns out my guess was right, 15 minutes later rsync stopped
all by itself when it ran out of space on the mount point drive.
And this is not the first time I have copied things to a drive that I
thought was mounted, but was not. It's a perennial problem, and it's
high time I figured out a way to keep it from happening.

If you are using/allowing the system to automagically mount your device
you should test if it is there before starting the backup.

I used this method years ago to back up my club accounting data.

OOPS ... wrong! I was pushing the files to my ownCloud server so I used
ping to test for it's existence.

Here is the script (obfuscated to protect the guilty.) You could figure
out your own test.

#!/bin/bash

/bin/ping -c 1 cloud.host.net 2> /dev/null | grep " 0% packet loss" >
/dev/null

if [[ $? == 0 ]]; then
echo cloud found
rsync -n -avz --update /home/user/Data/FILE /user/ownCloud/CLUB/DATA
else
echo cloud not found
fi


\\||/
Rod