Otdfbt

I am trying to move several TB from multiple hosts to other hosts. Each host will have 8 different rsync commands specifying different directory structures and I will be replicating this across nine different hosts.

Below is an example of what I'll be running on each host (different destinations but identical structures):

sudo -u svc_unix timeout 43200 rsync -azve ssh /tmp01/var/lib/data/engine_* --exclude 'db1' --exclude 'db2' 10.2.2.16:/tmp/prod/RSYNC_TEMP
sudo -u svc_unix timeout 43200 rsync -azve ssh /tmp01/var/lib/data/client --exclude 'db1' --exclude 'db2' 10.2.2.16:/tmp/prod/RSYNC_TEMP
sudo -u svc_unix timeout 43200 rsync -azve ssh /tmp01/var/lib/data/logs --exclude 'db1' --exclude 'db2' 10.2.2.16:/tmp/prod/RSYNC_TEMP
sudo -u svc_unix timeout 43200 rsync -azve ssh /tmp01/var/lib/data/backup* --exclude 'db1' --exclude 'db2' 10.2.2.16:/tmp/prod/RSYNC_TEMP

sudo -u svc_unix timeout 43200 rsync -azve ssh /tmp01/var/lib/data/engine_* --exclude 'db' 10.2.2.16:/tmp/QA/RSYNC_TEMP
sudo -u svc_unix timeout 43200 rsync -azve ssh /tmp01/var/lib/data/client --exclude 'db' 10.2.2.16:/tmp/QA/RSYNC_TEMP
sudo -u svc_unix timeout 43200 rsync -azve ssh /tmp01/var/lib/data/logs --exclude 'db' 10.2.2.16:/tmp/QA/RSYNC_TEMP
sudo -u svc_unix timeout 43200 rsync -azve ssh /tmp01/QA/backup* --exclude 'db' 10.2.2.16:/tmp/QA/RSYNC_TEMP

We plan on running this manually every night from 5pm - 5am (12 hour timeout) until it's synced to the new hardware, with a final run done to account for any changes for when we cut over officially.

Obviously, I'd rather not manually start up 72 different rsync jobs every night, and I'd like to figure out a simple way of killing them early if necessary.

Does anyone have any ideas on what would be a good way to manage this task?

Are you looking for a way to script this? Will something like this work (issued from the destination with sudo -u svc_unix timeout 43200 preceding the script - also remove the echo syntax around the rsync command in line 17)?

#!/bin/bash

#Configs:
ip=("10.1.1.1" "10.1.1.2" "10.1.1.3" "10.1.1.4" "10.1.1.5" "10.1.1.6" "10.1.1.7" "10.1.1.8")
source_base="/tmp01/var/lib/data"
source_array=(engine client logs backup)
destination_base="/tmp"
destination="RSYNC_TEMP"
## Just match the elements of the array with the IP addresses in $ip[]
destination_array=(prod QA foo bar stuff stuff prod dev)
exclude_file="some.file"

echo "script start time: $(date)"

for i in $(seq 1 $#ip[*]); do
 echo " start time for $ip[$((i -1))]: $(date)"
 echo "rsync -azqe ssh $ip[$((i -1))]:$source_base/$source_array[0]_* :$source_base/$source_array[1] :$source_base/$source_array[2] :$source_base/$source_array[3] --exclude-from=$exclude_file localhost:$destination_base/$destination_array[$((i -1))]/$destination/"
 echo "done with files coming from $ip[$((i -1))]"
 echo "script end time: $(date)"
done