Tips on using the rsync command. rsync performs incremental filesystem transfers, allowing filesystem duplication or snap shotting. Alternatives to rsync on Unix systems include cp -r, pipes between tar commands, or unison.
rsync behaves differently if the source directory has a trailing slash. Study and learn the difference between the following two commands before moving on. Use the -n option to rsync when testing to preview what would happen.
$ rsync -n -av /tmp .
$ rsync -n -av /tmp/ .
Normally, the -a option can be used to perfectly mirror the files. However, if the target filesystem does not support permissions, a different set of options should be used to avoid warnings from rsync. To synchronize data to a USB drive with a FAT filesystem, I use the -rlt options.
#!/bin/sh
RSYNC="rsync --size-only --delete --delete-excluded --exclude-from=~/.rsync/exclude -rlt"
TARGET=$1
$RSYNC ~/Talks $TARGET
mkdir -p $TARGET/backup/repository
$RSYNC ~/share/repository/ $TARGET/backup/repository
A ~/.rsync/exclude can be used to list common file patterns to ignore, for example .DS_Store files on Mac OS X.
$ cat ~/.rsync/exclude
.DS_Store
.FBCLockFolder
Set the -e option of rsync to use ssh instead of rsh when copying to a remote system. While ssh is slower than rsh, the data being transfered will be encrypted.
$ rsync -e 'ssh -ax' -avz example.org:/tmp .
If speed is a concern, use a weaker encryption option to ssh.
$ rsync -e 'ssh -c blowfish -ax' -avz example.org:/tmp .
The -ax options to ssh disable Secure Shell (SSH) agent and X11 forwarding, which are not needed by rsync. Also consider setting -o ClearAllForwardings to ssh, to prevent possible automatic port forwardings. For more information on options to OpenSSH, see ssh(1) and ssh_config(5).
To avoid stalls, set the --timeout option, though not low enough that rsync times out before it can build the filesystem differences in memory. In rare cases I have seen rsync not exit, so for unattended runs like filesystem snapshots I set the --timeout option to ensure the command will eventually quit.
Notes on how to configure periodic rsync runs over SSH. Filesystem duplication or snap shotting scripts may set the following up in different ways; the method outlined here mirrors the home directory of the user running the script from a client system to a backup server.
- Setup a SSH key pair without password.
A public key without a password allows unattended periodic backups. The public key should be locked down to only allow backups on the system the rsync is done to. These notes are for OpenSSH as of version 3.8.
On the system the rsync backup script is run on (client in these notes), create a SSH keypair.
client$ ssh-keygen -N '' -C backup1 -t rsa -f ~/.ssh/backup
- Configure public key on backup server.
On the system the rsync backup script connects to (server in these notes), configure the public key. These notes cover OpenSSH; consult the manual if a different SSH implementation is being used. Details on common problems with OpenSSH public key authentication.
client$ scp ~/.ssh/backup.pub server:.ssh
server$ cd ~/.ssh
server$ cat backup.pub >> authorized_keys
server$ rm backup.pub
For security, the authorized_keys file should be edited to only allow specific commands among other limitations. For more information, see sshd(8) and sshd_config(5). The command limitation to use can be determined by running the rsync with the -e 'ssh -v -v -v' option to see the exact command run on the server.
The following example shows how to restrict a public key in the authorized_keys file to only run the specified command, along with other restrictions on the connection. The limitations must be listed on one line, prior to the lengthy public key data.
command="rsync --server -v --timeout=999 --delete-excluded . backup/client",21B5;
no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3Nza2026;
- Create the target backup directory on the server.
rsync will not create the target directory ($HOME/backup/client) on the server; the target directory must be manually created.
server$ mkdir -p ~/backup/client
- Create backup script dobackup.
Use the example script linked to, and localize it as needed.
To test the script, try prefixing the rsync call with echo to see what would be run, or add the -n option to rsync to see what it would copy.
- Configure a ~/.rsync/exclude file to list files not to backup.
Cache files and other transitory data should be skipped. For information on how to exclude files, see EXCLUDE PATTERNS in rsync(1).
$ cat ~/.rsync/exclude
.DS_Store
.mozilla/**/Cache
*.o
- Run backup script via a crontab(5) entry.
In addition to automatic runs, the script can be run manually from the command line.
client$ crontab -l
@daily $HOME/bin/dobackup
client$ ~/bin/dobackup
2026;
Need to figure out simple locking to prevent an automated run from conflicting with a manual run.
Mac OS X 10.4 (Tiger) supports the -E option to rsync, which copies extended filesystem attributes.
On previous versions of OS X, compile rsync with the rsync+hfsmode patch. Note that rsync may have trouble with symbolic links (ownerships and permissions) and sockets (perhaps -gHlopqrtx instead of -a).
Linux/Mac OS X: Allow ssh access to server without requiring password.
1) Generate a new key (the -f command -f ~/.ssh/access gives it an optional name, -C allows for a comment):
xtian@desktop$ ssh-keygen -C commont_or_purpose -d
Generating public/private dsa key pair.
Enter file in which to save the key (/home/xtian/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/xtian/.ssh/id_dsa.
Your public key has been saved in /home/xtian/.ssh/id_dsa.pub.
The key fingerprint is:
33:3c:5c:41:98:1b:fc:f5:9e:69:56:2e:0b:f1:24:7f xtian@desktop
* The -d option specifies DSA keys (instead of RSA keys). The ssh v2 protocol uses DSA keys, and is widely regarded as more secure than v1.
* After entering the command, hit enter three times (to take the default filename, and to enter no passphrase.)
* Congratulations. Your public and private keys are now saved to ~/.ssh/id_dsa.pub and ~/.ssh/id_dsa, respectively.
2) Copy the key to your Web server:
xtian@desktop$ scp ~/.ssh/id_dsa.pub www.stonescape.net:.ssh/authorized_keys2
* At this point, if you've never used ssh from your OS X box before, you'll be prompted to verify the fingerprint of the server's key. Answering "yes" will save the server's fingerprint in a local cache. Should the fingerprint ever change, ssh (and scp) will sound an alarm, as this could be an indication of a man-in-the-middle attack in progress.
* You will be prompted for your password on the Web server. Enter it, and the key file will be copied.
3) Modify the key to your Web server:
xtian@desktop$ ssh stonescape.net
xtian@server$ cd .ssh
xtian@server$ cat id_dsa.pub >> authorized_keys2
xtian@server$ rm backup.pub
xtian@server$ exit
* You will be prompted for your password on the Web server.
4) Test the ssh key:
xtian@stonescape$ ssh www.stonescape.net
login: Mon Oct 29 10:58:32 2001 from desktop.stonescape.com xtian@server$
* It should log you in without a password. If not, check your work. Also check that your Web server allows public key exchange (it's on by default, and is rarely disabled. Check with your friendly local sysadmin if you're not sure.)
Posted by xtian
@ 11:29 PM EST
Restricting access to a .ssh key for a specific command
The following example shows how to restrict a public key in the authorized_keys (.ssh directory) file to only run the specified command, along with other restrictions on the connection. The limitations must be listed on one line, prior to the lengthy public key data.
command="rsync --server -v --timeout=999 --delete-excluded . backup/client",↵
no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3Nza…
Posted by xtian
@ 06:44 PM EST
Configuring Mail.app to use SSL without the annoying "The root certificate for this server could not be verified" warning.
Mail asks if you want to accept an SSL certificate for each IMAP account. This happens each time Mail opens, even if you click Remember My Decision.
Solution
1. Click "Show certificate" when Mail asks if you want to accept the certificate.
2. Press the Option key while dragging the certificate to the desktop. The certificate's icon appears on the desktop.
3. Add the certificate to your keychain by dragging its icon on top of Keychain Access. Tip: Keychain Access is located in the Utilities folder (/Applications/Utilities).
4. When Keychain Access opens, you may be prompted to select which keychain to import to. If this happens, choose a keychain and also select X509 Anchors.
Additional information
Mail will continue to ask if you want to accept an SSL certificate each time it opens if the certificate is an expired server certificate or is signed by an unknown certificate authority.
Posted by xtian
@ 06:23 PM EST
