Syncing files between two computers with OS X 10.6, using rsync & SSH
If you have been working on a specific computer for a while, and need the files in another location it can be a bit of a pain at times to make sure that everything is transferred properly. This Friday when I got home after purchasing the OS X 10.6 Snow Leopard upgrade I had a chance to implement some file syncing between my MacBookPro and iMac. Since I wanted to format both computers at the same time so that they would have as clean as possible of an install, it was the perfect time to do some research on how I could get some good file synching between my two main computers. Looking around on the internet there is a lot of posts on the subject, and a lot of people talking about a lot of different ways to accomplish file syncing. The main problem I found were that people were recommending to use programs like DropBox, SugarSync, or ZumoDrive. Don’t get me wrong, these are all are great programs (and I do currently have a ZumoDrive account which I use for files I know I will need at home on my macs or the office pc), they all involve subscription fees if you wish to use anything over the trial account storage limit. After looking through some of the options I stumbled upon a post talking about a “hidden” feature in OS X. Well the feature isn’t really hidden, but it is only available through terminal. This feature is rsync. rsync is a unix tool that can be used to synchronize files and directories from one location to another, and this includes the ability to do so over a SSH connection. After finding out about the SSH ability I knew that this was the direction for me to head in. rsync would allow to synchronize as much data as I wanted, but it wouldn’t allow me to have a fancy GUI or iPhone app. Those were things that I was willing to live with out, until it bothers me and I try make my own.
Below are the steps that I used to get rsync working between my MacBookPro and my iMac:
- On both computers, open up the preference menu and go into the Sharing section. Make sure that both the File Sharing, and Remote Login boxes are checked. Be sure to take note of the Computer Name, as we will be using this later to make our connection via SSH(using the computer name will limit you to syncing on your local network, in order to SSH from anywhere over the internet you will need to use you ip and have the proper ports forwarded to proper computers).
- If you have not done so already for other reason you will need to generate a Key Pair on each computer for the SSH connection. Follow the information on this site very carefully to ensure that you generate the Key Pair properly. If this is done incorrectly the SSH connection will not work.
- Now that you have the ability to connect via SSH between the two computers, it is time to get down to using rsync.
I have set up both my MacBookPro and iMac to have a set of files with the same file names that will be synced. This makes it easier to keep track of, but you can sync to and from any folder you wish. I recommend setting up a test folder at first to make sure everything works properly before dealing with important data. In order to sync data between your two computers you will need to use a command similar to this(I say similar because you may wish to change the which rsync is running, and it will change if you are syncing to or from a computer): - If you are successful in connecting over SSH and syncing data to or from your computer, you should now create a bash script file that you can run and sync up all of your important files at once. I have created 2 files for each computer. One if for syncing to the other computer and one if for syncing from the other computer. In each file list out the commands that you would like to run and save it as sync_to or sync_from. Below is an example of one of my sync_to file that is used to sync from my MacBookPro to the iMac.Im sure if you are a bit better with bash, or if I look into it a bit further, that this can be done all in one file.
- Now to sync the files between your computers all you need to do is navigate to the directory containing your bash script, and run it.
eg:
1 2 3 4 | #sync from this computer to another computer rsync -avzr --delete /folder_to_sync_from/ -e ssh user_name_used_in_setup@ip_or_computer_name_used_in_setup:/folder_to_sync_to/ #sync from another computer to this computer rsync -avzr --delete -e ssh user_name_used_in_setup@ip_or_computer_name_used_in_setup:/folder_to_sync_to/ /folder_to_sync_from/ |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | #!/bin/bash #REPLACE user_name_used_in_setup WITH THE USER NAME USED DURING SSH KEY PAIR SETUP #REPLACE iMac WITH THE COMPUTER NAME OR IP USED DURING SSH KEY PAIR SETUP echo ">> starting sync to iMac" #sync the courses folder rsync -avzr --delete /courses/ -e ssh user_name_used_in_setup@iMac:/courses/ #sync the flash_workspace folder rsync -avzr --delete /flash_workspace/ -e ssh user_name_used_in_setup@iMac:/flash_workspace/ #sync the iPhone_workspace folder rsync -avzr --delete /iPhone_workspace/ -e ssh user_name_used_in_setup@iMac:/iPhone_workspace/ #sync the personal_as3_classes folder rsync -avzr --delete /personal_as3_classes/ -e ssh user_name_used_in_setup@iMac:/personal_as3_classes/ #sync the processing_workspace folder rsync -avzr --delete /processing_workspace/ -e ssh user_name_used_in_setup@iMac:/processing_workspace/ #sync the open_frameworks folder rsync -avzr --delete /open_frameworks/ -e ssh user_name_used_in_setup@iMac:/open_frameworks/ #sync the open_frameworks folder rsync -avzr --delete /bash_scripts/ -e ssh user_name_used_in_setup@iMac:/bash_scripts/ echo ">> finished sync to iMac" |
1 2 | cd /bash_scripts/ ./sync_to_imac.sh |
This method of syncing seems to be working out pretty well for me, although it is limited to my local network the way I set up the SSH Key Pairs. Maybe in the future I will look into setting it up using ip’s and the properly forwarded ports but for now it is great! If you are reading this and happen to know a few tips or tricks that can be used with rsync, please comment below and let me know!
Tagged as Bash, file syncing, OS X, os x 10.6, rsync, SSH Key Pair, sync + Categorized as Bash
You may consider upgrading the version of rsync you use to sync files, because the version Apple include doesn’t contain certain fixes & patches that help keep important attributes on the files.
Mike Bombich (CarbonCopyCloner) has a guide that will give you a patched mac specific version 3.0.6 compared to the almost 3 year old 2.6.9 that Apple supplies with 10.6.
http://www.bombich.com/mactips/rsync.html
He also uses other flags that will help preserve data such as creation times, ACL’s, Extended attributes, resource forks etc.
It is worth upgrading at both ends of the syncing too, incase you find you need to copy in the other direction in the future.
Hey Drew,
Thanks a lot! I will take a look into it.
So I’ve been working on a solution like this…
Isn’t –delete a bit dangerous? It gets rid of all “extraneous” files on the destination. If you happen to, just once, run it the “wrong way”, you’ve just deleted your only copy of something.
Why not instead use -u (–update, skip files that are newer on the receiver) and run it both directions. That way you get a bidirectional sync, with the newest files “winning”, no matter which machine they’re on. (You’d still need -a and want -z. -r is implied by -a.)
Just $0.02. If you have a machine you _want_ to be the “winner”, then –delete makes more sense. But for me, I often work on both machines and need to be sure to preserve newest.
Hey Musikia,
Thanks for the information! This was my first time using rsync and I guess that I had overlooked that option. I will most likely be changing my scripts to use this in the future, and will update the blog post when I get a chance.
- Mike