Monday, 8 July 2013

rsync - file transfer program for *NIX systems

Long back when I wrote an script to get backup copy of the file, I used "cp" command. Few of my colleagues recommended me to get through "rsync". On my study I found this rsync is a powerful utility software and network protocol for *NIX based systems that synchronizes files and directories from one location to another while minimizing data transfer of files and directories from one location to another using data differencing when appropriate.

Instead of having the entire file to copy, rsync copies only the differences of files that have actually changed and transferred to the destination rather than the whole file.  This makes updates faster, especially over slower links like modems.

Differences are compressed on the fly, further saving you file transfer time and reducing the load on the network.

rsync is used as a backup and mirroring tool.

I have customized my tools to backup my scripts from staging to production servers automatically through cron.

Some of the features of rsync are as below

- Support for copying links, devices, owners, groups and permissions.
- Exclude and exclude-from options similar to GNU tar.
- Does not require root privileges.
- Pipe-lining of file transfers to minimize latency costs.

Discussed below are few of the regular commands which are used very frequently.

Before understanding the below, I would suggest you to go through the man page for the options.

I have two servers, I would name it as below..
1. Source
2. Destination

Trust between by source and the destination servers are through host-key authentications. 

If you need to deploy host key authentication here is the link

Objective: Reference guide for rsync.

Since I would have a constant backup of my scripts, I have defined my path to store the files in the destination server.

On the destination host, configure your rsync

#yum -y install rsync xinetd
#vi /etc/xinetd.d/rsync
disable = no
#/etc/init.d/xinetd start

#lsof -i :873
#mkdir -p /backup/{scripts,configs,logs}
#vi /etc/rsyncd.conf

#cat /etc/rsyncd.conf

hosts allow = <IPADDR of your source server>
hosts deny = *
list = true
uid = root
gid = root
read only = false

Source - rsync one liners 

# rsync file1 /tmp - copies file1 to /tmp.
# rsync -t file1 /tmp -  copies and preserves modification time.

Connecting to rsync daemon 

you can connect to rsync daemon using double colon :: instead of a single colon to separate the host name from the path.

# rsync

# rsync -tv *.sh -> transfer all files matching the pattern to the directory src of the destination.
# rsync -avz ~/Documents/scripts/ -> copies all the files inside the directory.
# rsync -avz ~/Documents/scripts -> copies complete directory to the destination.
# rsync -avz --exclude-from=/etc/rsync_exclude.lst ~/Documents/scripts> excludes the files in the /etc/rsync_exclude.lst
# rsync -azvh --inplace ~/Documents/scripts -> update files at the receiver end in human readable format.
# rsync -avzi root@ . -> view the difference in the files or directories between source and destination.

> specifies that a file is being transferred to the local host.
f represents that it is a file.
s represents size changes are there.
t represents time-stamp changes are there.
o owner changed
g group changed.

NOTE: You must be very careful while deleting files using rsync command.
I would suggest to use an dry-run before executing the rsync to delete files at the receiver end.

# rsync -av --delete --dry-run -i ~/Documents/scripts
building file list ... done
*deleting   scripts/
*deleting   scripts/
sent 5192 bytes  received 22 bytes  10428.00 bytes/sec
total size is 398198  speedup is 76.37 (DRY RUN)

Once you confirm the above are the files which are not in the source, then continue to remove the dry-run and execute the command, which deleted files which are not in the destination server. 

Now, your source and the destination servers are in sync.