How I Use Amazon S3

This post suggestion has been sitting in my Skribit account for a while so I thought I would finally address it. S3 is Amazon's developer-aimed online storage solution. In recent years, consumer-friendly applications and tools have added support for Amazon S3, making it a viable backup solution for anyone, especially those with a fast Internet connection. The price is right too: cheap.

I won't go through the details of getting started with S3 as I briefly mentioned that in my S3 server backup post.

What Stays and What Goes: Relying on the Cloud

As a general rule of thumb, if I have worked on a file in the last 10 days or so I keep it on my machine and on S3. If I don't use it that often, it goes to my S3 account and I delete it from my computer. I've gotten into the habit of manually putting files into S3 with Transmit.
Panic Transmit S3 Folders

I have most of my files in the same S3 bucket and within that bucket I have a few folders to organize my documents.

JungleDisk rsync s3 PDFs folder

I'll be the first to say this is not the most efficient method. Launching a separate app to transfer files breaks up the user experience from the seamless action that it should be. I imagine that most people would prefer using something like JungleDisk and its automatic backup feature. Once JungleDisk is fired up, your S3 account is accessible through a regular Explorer/Finder/Nautilus window.

JungleDisk Backup

Other Uses

After a comment by lead developer for WordPress, Mark Jaquith, I've additionally started backing up my home directory (/Users/Paul) with rsync and JungleDisk. While it is possible to backup my home directory entirely with JungleDisk, I'm more accustomed to having control with rsync. I usually run a command like this after JungleDisk is active:

rsync -avvz --size-only --delete --exclude .DS_Store --exclude .Trash --exclude .svn --exclude Library/Caches --exclude Library/Mail/ --exclude Library/Application\ Support/Inquisitor/IconsA/  --exclude Library/Logs --exclude Library/Application\ Support/SyncServices/ /Users/Paul /Volumes/JungleDisk/

That command has many excludes to make the entire rsync quicker. Transferring a few large files is a lot faster than transferring many small files due to opening and closing connections, latency and the like. There's no need for me to backup my inbox since it is already on the mail server with IMAP. If you end up running rsync often, it's worth setting up an excludes file.

JungleDisk rsync s3 home directory

Backing up your entire home directory, including music, will take a long time during the first run regardless of the connection you have. Transferring many small files such as Adium chat logs takes time. For that reason, most people will be better off using S3 for important documents and leaving larger backups to an external hard drive and imaging software like Carbon Copy Cloner or Super Duper.

Rsync backup music jungledisk
Example rsync backing up music to S3 via JungleDisk. Bandwagon is easier to work with though.

Do you use Amazon S3? How so? All of that being said, if my laptop (my only computer, except for a window manager-less media server) were stolen right now I wouldn't lose enough data to worry about. Most everything I could possibly need is on Amazon S3, Gmail and a hosted SVN account.