Friday, February 18, 2022

[SOLVED] rsync: remote sync file‘s create-time(became to 1970) is different from local file's creation time

Issue

I used RSYNC to upload a folder from local host to remote host. It was set to a scheduled task by CRONTAB. And an error occurred during uploading: a file's (in folder) transmission failed, and its creation time became to 1970/01/01(obviously incorrect). Also, the folder's(on remote host) creation time is different from local folder's(on local host) creation time. The former is a few minutes later than the latter. Here is my code(which is set to be a scheduled task by crontab):

time=$(date "+%Y-%m-%d")
echo $time
count_tmp=0
while [ $count_tmp -le 4 ]
do

/usr/bin/expect <<-EOF
spawn rsync -avzP -e "ssh -p xxx" --bwlimit=5000 $time xxx@xxxx:/xxx/xxx
set timeout 300
expect {
 "*yes/no" { send "yes\r"; exp_continue }
 "*password:" { send "changeme\n" }

}
send "exit\r"
expect eof
EOF

if [ $? -ne 0 ]
then
 let count_tmp++
 echo "failed, try again"
 sleep 1h
else
   echo "success"
   break
fi
done

And following is : 2020-08-01 is the folder, and xxx-2020-08-01.txt are files in it. records_history_information-2020-08-01.txt is uploading when error occurred and process interrupted.

-remote host folder's(2020-08-01) info

drwxr-xr-x 2 hgd hgd 4096 Aug  1 20:25 2020-08-01

-remote host file's(records_history_information-2020-08-01.txt) info

-rw-rw-rw- 1 hgd hgd  58182912 Jan  1  1970 records_history_information-2020-08-01.txt

-local host folder's(2020-08-01) info

drwxr-xr-x 2 root root       4096 Aug  1 20:16 2020-08-01

-local host file's(records_history_information-2020-08-01.txt) info

-rw-rw-rw- 1 mysql mysql 727396510 Aug  1 20:16 records_history_information-2020-08-01.txt

local crontab log(show records_history_information-2020-08-01.txt upload interrupted, next is 2020-08-02)

2020-08-01/records_history_information-2020-08-01.txt
2020-08-0292   8%   13.28MB/s    0:00:48

I have two questions:

-The records_history_information-2020-08-01.txt's incorrect creation time(which is 1970) is the result of its fail during uploading?

-The host never shuts down (both) and I have set 5-times re-upload. Why does it close uploading? Could it be caused by the timeout of Rsync? Or other reason?


Solution

January 1st, 1970 is the start of the Unix "epoch" (i.e. time zero), so seeing that in the case of a failure sounds plausible. Rerunning rsync with -t or -a should fix it.

Your expect script has a 5 minute timeout. If that's insufficient time to complete the transfer, or if there's an extra user prompt you haven't anticipated, then yes it will interrupt the upload.

I suggest you use a higher timeout value for the "expect eof".

A few other notes:

  • The "send exit" seems unnecessary.
  • The "yes/no" thing looks like a bad idea; if ssh doesn't know the host then something fishy is going on, and you probably shouldn't be connecting blindly.
  • Embedding a password in a script is not ideal; you should set up an ssh key for automatic log in (see ssh-keygen and ssh-copy-id). This will eliminate the need for expect completely (you can use the timeout tool instead).


Answered By - ams
Answer Checked By - Cary Denson (WPSolving Admin)
at