Category Archives: Backups

Keeping Mobile Avid Media Updated

On most of the films that I’ve worked on, I’ve needed to keep at least two Avid systems up to date. This could be because an editor wants to be able to cut from home, or the Director wants the ability to cut on set. To handle this I wrote a Terminal (bash) script that searches for any new MXF or OMF media on specified volumes since the last time the script was run. On first-run, the script does nothing but set the date to compare against the next time the script is run, though if you need to change that instructions are below.

Automating Avid Project Backups

REVISED 9/23/09: Rewrote for Leopard

The Importance of Backing Up

The most important part of an Avid project is, of course, the project directory. Media can be replaced, albeit tediously, but your project directory cannot. Therefore, it is not only important that you back up your project daily, but that you also take it with you when you leave for the night. Simply copying the project from one workspace to another is not sufficient, nor is just copying it to your local Avid drive. If a disk or two fails on the Unity and the only copy of the project you have is there, you’re probably screwed. If there’s a fire and both your Unity and your desktop Avid burn up, you’re definitely screwed. The number one rule of making backups is that the two copies must be geographically separated. I cannot stress that enough.

Thankfully, it’s easy and painless to do this, and you can automate it. It doesn’t matter whether you use a PC or Mac, you can automate it on either platform. But for the purposes of this tutorial, and since I’m pretty solidly in the Mac user base now, I like to use Automator and Growl (both Mac-only). To download my Automator script, click the link at the bottom of this post.

Automator

The process of creating an Automator workflow to back up your Avid project is fairly simple. The process I’ve chosen involves syncing the Avid project directory on the Unity to a copy of it on your Avid’s desktop, creating a zip file of the desktop copy, and then transferring that to a USB drive. It requires five actions, which are these:

  1. Run a Shell Script

    Don't forget to change this command to your own project directory and desktop folders!

    • This command runs the program rsync, with the arguments “-avP” to provide directory recursion (so it will copy all subdirectories of your project directory), copy almost everything while maintaining file attributes, and be verbose about what it’s doing (which is good if you run this command manually from the Terminal, but won’t have an affect in Automator). The command is as follows:
    • rsync -avP --delete --exclude "*.log" --exclude "~avid_remove*" --exclude ".DS_Store" --delete-excluded /Volumes/Project/YOUR_PROJECT /Users/YOUR_USER/Desktop
    • The “–delete” argument tells rsync to delete any files on your local directory that do not exist on the Unity. So if you delete a bin on the Unity, it will make sure that bin is removed from your local drive, too.
    • The “–exclude” arguments tell rsync NOT to back up any files that match *.log, ~avid_remove, or .DS_Store in their filenames
    • The –delete-excluded argument tells rsync to delete any files in your backup directory that do not exist in your project folder, including files that have been excluded above, if for some reason there are any in your backup directory
    • The reason I use rsync instead of just re-copying the entire Avid project folder to the desktop and overwriting yesterday’s copy is that when your project directory gets to be several GB in size, it is much quicker and just as thorough to copy only the files that have changed since your last backup. Otherwise you end up recopying a bunch of bins that haven’t changed since months ago, and that takes time.
  2. Get Specified Finder Items
    • Once rsync finishes, this command runs to tell Automator where your backup directory is.Get Finder Items (Leopard)
  3. Create Archive
    • This action takes your desktop Project directory, and creates a zip file out of it. You want to do this because bin files (.avb) are highly compressible, meaning that you can take your 2GB project directory and compress it down to a couple hundred MBs.
    • In the “Save as:” text box, you can see I’m telling Automator that I want my zip file to be named with a variable for today’s date, such as “Warrior.90922.zip” and that I want it saved to the Desktop.

    Create Archive (Leopard)

    My Automator variable that inserts today's date. Rearrange the parts of the date however you like. I choose this method so Finder sorts my backups chronologically.

  4. Copy Finder Items
    • This action takes your newly minted zip file and copies it to your flash drive.
    • N.B. Plug your flash drive in before running this workflow. If you don’t, Automator will fail, but if you do then you can leave and come back in 10 minutes to a fully backed up project already on your flash drive.
    • Copy to Flash Drive (Leopard)

  5. Show Growl Notification (optional)
    • This step allows you to display a Growl notification once the backup has completed. I find this useful because I can go make some tea while Automator is running and see at a glance when I come back if it’s done yet. Growl will also make a sound when it displays the notification, so you can grab your flash drive the instant it’s done and go home. Growl is a separate application you have to install, though, so there is a bit of extra prep you need to do before this action will be available inside Automator.

      Growl (Leopard)

      Make your Growl notification "Sticky" if you want the notification to remain on screen until you click on it.

And that’s it. After Automator finishes, you’ll have a backup copy of your project on both your local Avid drive, and your flash drive. You should then take your flash drive home with you every night, so that your backup copies are in separate places. I also keep an archive of every zip file I make on an external hard drive. Most of the time you won’t need this, but sometimes it’s useful to be able to go back a few days or months in your project’s history.

Attachments:

Backing Up Unity Media

Every once in a while it becomes necessary to back up all of the media on your Unity, and this is a fairly big task. It can be terabytes of data that you need to transfer to a backup drive or drives, and there are better ways of doing this than just drag and drop. So whether the backup is needed at the end of the show for archival purposes, or somewhere in the middle because you’re switching cities and need to ship the Unity to a new office, here’s a little script I wrote to help accomplish this task safely and reliably. Some knowledge of Terminal is required (but not too much).

Backup Strategies

The script I’ve detailed here does rely on a bit of manual work. You will need to do the math to figure out which and how many workspaces from your Unity will fit onto the external drive(s) you have available. The Unity Administration Tool should help you out with that.

This script should be modified and run once per external drive. So once you know how you want to distribute your Unity workspaces across your external drives, then you should only set this script up for one volume at a time. When it finishes and is successful, change the Target Volume and Source Directories in the script file and execute it again.