Automating Avid Project Backups

REVISED 9/23/09: Rewrote for Leopard

The Importance of Backing Up

The most important part of an Avid project is, of course, the project directory. Media can be replaced, albeit tediously, but your project directory cannot. Therefore, it is not only important that you back up your project daily, but that you also take it with you when you leave for the night. Simply copying the project from one workspace to another is not sufficient, nor is just copying it to your local Avid drive. If a disk or two fails on the Unity and the only copy of the project you have is there, you’re probably screwed. If there’s a fire and both your Unity and your desktop Avid burn up, you’re definitely screwed. The number one rule of making backups is that the two copies must be geographically separated. I cannot stress that enough.

Thankfully, it’s easy and painless to do this, and you can automate it. It doesn’t matter whether you use a PC or Mac, you can automate it on either platform. But for the purposes of this tutorial, and since I’m pretty solidly in the Mac user base now, I like to use Automator and Growl (both Mac-only). To download my Automator script, click the link at the bottom of this post.

Automator

The process of creating an Automator workflow to back up your Avid project is fairly simple. The process I’ve chosen involves syncing the Avid project directory on the Unity to a copy of it on your Avid’s desktop, creating a zip file of the desktop copy, and then transferring that to a USB drive. It requires five actions, which are these:

  1. Run a Shell Script

    Don't forget to change this command to your own project directory and desktop folders!

    • This command runs the program rsync, with the arguments “-avP” to provide directory recursion (so it will copy all subdirectories of your project directory), copy almost everything while maintaining file attributes, and be verbose about what it’s doing (which is good if you run this command manually from the Terminal, but won’t have an affect in Automator). The command is as follows:
    • rsync -avP --delete --exclude "*.log" --exclude "~avid_remove*" --exclude ".DS_Store" --delete-excluded /Volumes/Project/YOUR_PROJECT /Users/YOUR_USER/Desktop
    • The “–delete” argument tells rsync to delete any files on your local directory that do not exist on the Unity. So if you delete a bin on the Unity, it will make sure that bin is removed from your local drive, too.
    • The “–exclude” arguments tell rsync NOT to back up any files that match *.log, ~avid_remove, or .DS_Store in their filenames
    • The –delete-excluded argument tells rsync to delete any files in your backup directory that do not exist in your project folder, including files that have been excluded above, if for some reason there are any in your backup directory
    • The reason I use rsync instead of just re-copying the entire Avid project folder to the desktop and overwriting yesterday’s copy is that when your project directory gets to be several GB in size, it is much quicker and just as thorough to copy only the files that have changed since your last backup. Otherwise you end up recopying a bunch of bins that haven’t changed since months ago, and that takes time.
  2. Get Specified Finder Items
    • Once rsync finishes, this command runs to tell Automator where your backup directory is.Get Finder Items (Leopard)
  3. Create Archive
    • This action takes your desktop Project directory, and creates a zip file out of it. You want to do this because bin files (.avb) are highly compressible, meaning that you can take your 2GB project directory and compress it down to a couple hundred MBs.
    • In the “Save as:” text box, you can see I’m telling Automator that I want my zip file to be named with a variable for today’s date, such as “Warrior.90922.zip” and that I want it saved to the Desktop.

    Create Archive (Leopard)

    My Automator variable that inserts today's date. Rearrange the parts of the date however you like. I choose this method so Finder sorts my backups chronologically.

  4. Copy Finder Items
    • This action takes your newly minted zip file and copies it to your flash drive.
    • N.B. Plug your flash drive in before running this workflow. If you don’t, Automator will fail, but if you do then you can leave and come back in 10 minutes to a fully backed up project already on your flash drive.
    • Copy to Flash Drive (Leopard)

  5. Show Growl Notification (optional)
    • This step allows you to display a Growl notification once the backup has completed. I find this useful because I can go make some tea while Automator is running and see at a glance when I come back if it’s done yet. Growl will also make a sound when it displays the notification, so you can grab your flash drive the instant it’s done and go home. Growl is a separate application you have to install, though, so there is a bit of extra prep you need to do before this action will be available inside Automator.

      Growl (Leopard)

      Make your Growl notification "Sticky" if you want the notification to remain on screen until you click on it.

And that’s it. After Automator finishes, you’ll have a backup copy of your project on both your local Avid drive, and your flash drive. You should then take your flash drive home with you every night, so that your backup copies are in separate places. I also keep an archive of every zip file I make on an external hard drive. Most of the time you won’t need this, but sometimes it’s useful to be able to go back a few days or months in your project’s history.

Attachments:

14 Comments

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

  1. Sounds like a great approach, but I’ve always bit the bullet and resorted to completely replacing my backups with an overnight transfer.

    I just don’t have a high enough comfort level with any automated sync utilities…especially since we keep our assets (renders, graphics, imports, etc) in subfolders within the Avid project directory.

    I’ve tested a few utilities that resulted in duplicate bins, files not getting replaced that I thought should, and so on. I chalk some of this up to my own ignorance, but nevertheless can’t seem to “let go” and let some open-source program handle the smart replacement of my precious project files.

  2. As it’s every assistant’s job to keep the project safe, I certainly understand wanting the peace of mind of using a process you’re comfortable with. The only assurance I can give you is that rsync is more than just a random open source program. It’s a very powerful utility that’s been on Unix systems for over a decade and is used by IT professionals worldwide to transfer files and make backups between servers. I’ve never had a problem with a file that wasn’t updated properly and have never looked for a file that should’ve been there and wasn’t. The options I’ve provided work for me in terms of giving me a mirrored copy of what’s on the server every time I run the rsync command, and I think they’d work in most Avid Project backup situations, although if you have different needs you’d need to adjust the options and arguments accordingly. The other thing I’d just note is that rsync doesn’t care what type of files it’s syncing, so whether you keep assets within the project directory or not is immaterial to the program. It will just find whatever is newer on the Unity than it is on your system and copy it over.

    If you’re open to giving it a shot you could really do both types of backups concurrently, and compare them to see if there’s anything you don’t like.

  3. Thanks so much for this, Evan. This saves me 20 minutes every day…

  4. You’re welcome! I always hope these things will save others as much time as they save me

  5. Just udpated the workflow to use a better rsync configuration. Also updated for Snow Leopard in the process.

  6. This is very useful. But one thing i also do is keep my project within my google drive or dropbox folder. Every time I save, it auto syncs the version in the cloud. If i go home and my edit suite burns to the ground, and on the way home I lose my pen drive, I still have my project!

  7. I do that, too, on small projects where I’m the only editor. On a big show the project has to live on the ISIS, and even if you make a symlink from your Dropbox folder the project just gets too big and changes so frequently that Dropboxing it isn’t really viable.

  8. Hi Evan,

    Thanks for sharing all this great information – Very kind of you – Your brain dump is incredibly enlightening and helpful!

    Regarding the process of automated project back-up, have you changed the process at all since 2009? Also, is there anything to bear in mind vis-a-vis implementing it on an Isis system?

  9. Nothing changes on ISIS vs Unity, that’s all the same. These days I just upload the zip files to cloud storage rather than copying to a flash drive. And I think given some effort one could create a backup system more like Time Machine, so rather than zip files you’d just have a repository of your project’s history and could more easily go back to a bin from a specific day, but I haven’t put any effort into figuring that out.

  10. Yeah, thats a great idea.

    One more question for ya. How would I edit the shell script to include all projects for the series Im currently working on? We have 11 different project – will have more as time goes on.

  11. If they’re all on one ‘projects’ workspace you can just make the rsync command something like:


    rsync -avP --delete --exclude "*.log" --exclude "~avid_remove*" --exclude ".DS_Store" --delete-excluded /Volumes/Projects /Users/YOUR_USER/Desktop

  12. Yes, they are all in one folder but the attic is in the same folder – id rather not bother keeping back up of it. Possible to remove it from the equation? Would it be –delete –exclude “Unity attic” ?

  13. Drew and Evan, I’ve been using this as a starting point for a while now. I also use

    –exclude “_SearchDB_*”

    to cut down on the file size, since the search database can be rebuilt anyway. This is especially helpful for multiple projects with lots of indexed SFX.

  14. Thank you for sharing this, how do you call “Today’s date”? It doesn’t seem to be a standard variable anymore…