Category Archives: Workflow

Experimenting with Streaming on a CDN

Experimenting with Streaming on a CDN

Recently I was asked to post a copy of the film I just finished cutting online for a producer in Europe to view. This meant I would need to encode a Flash Video file (FLV) or an H.264 MP4, and I would need to find someplace secure to host it online. On this and previous projects, I’ve often uploaded Quicktimes directly to my website and sent a link and login credentials to whoever needs them. If the recipients are in the United States, my hosting plan at MediaTemple is more than adequate to serve up a Quicktime or two. For slightly longer cuts I’ll usually go the extra step of encoding a FLV and directing the recipient to an HTML page with a flash player set up to play the FLV. I have also uploaded a 90-minute cut in FLV format to my server, but a couple people I sent it to had problems playing it all the way through, and when they had to reload the page they were forced to start downloading the 1.2GB file from the beginning.

In this case, since the producer is in Germany, I wanted to be able to provide him a quick, localized download, as well as the ability to jump to any part of the film instantly. This is a feature you find on most major video sites, and if your connection gets disrupted or you need to reload the page for whatever reason, you can pick up right where you left off with minimal downtime. Unfortunately I don’t have enough control of my MediaTemple server to install the appropriate software for this kind of streaming. Also, my server is located in Los Angeles, and having lived in Europe I’m well aware of how slow it is to transfer files transcontinentally. What I needed was a RTMP server in Europe, and without doing very much research I opted to try out Amazon Cloudfront.

The video below is an example hosted from my Cloudfront account. I didn’t have much I could put online publicly, so please enjoy the credits from a short film I cut called The Happiest Man Alive.

[jwplayer file=”HMA_Credits2.mp4″ streamer=”rtmp://s20fxl9ko2l7i5.cloudfront.net/cfx/st/mp4″ provider=”rtmp”]

Amazon Cloudfront

Amazon Cloudfront is an extension to their popular S3 file hosting service. The way S3 works is that you upload whatever files your website requires to Amazon S3’s servers, and then they take care of making sure that when someone requests your website, the appropriate files are transmitted from a server closest to the requestor. So for example, a user in Japan would download images on your website from an Amazon server in Asia, while in Los Angeles you’d download the same files from a server on the west coast. You pay a few cents per gigabyte of data transferred, but your website loads much faster for everyone worldwide since they’re not waiting to load it from a server half a world away. This is what a Content Delivery Network (CDN) is designed to do, and though most CDNs are designed for big corporations like Apple and Facebook, Amazon happens to run one designed and priced for individuals as well.

What Cloudfront does is add a few more localization controls to S3, as well as a streaming capability. There are other streaming services you can pay for, but for me what gave Cloudfront the edge was the combination of a CDN and streaming service. This way, not only can the producer stream my film in the same way he would watch a clip on Youtube, but it will stream to him from an Amazon server in Frankfurt, Germany, even though I uploaded the file to an Amazon server in the US.

Caveats

As usual, a few caveats.

1) Cloudfront and S3 are not replacements for a host server. They serve up files but they don’t serve up websites. You will still need your own server or a Google Site for your viewers to point their browser at, and you’ll need to configure a Flash player with the appropriate Cloudfront RTMP URL where your file is located.

2) If you don’t want random people seeing your content, make sure to protect the folder containing your HTML page in some way (.htaccess, server-side authentication, etc.). Your hosting provider likely provides at least basic folder password-protection.

3) Cloudfront does offer a “Private Content” option, which allows you to specify a security policy and expiration date in the link you send out to people, however I was not able to get this to work with a RTMP link in either Flowplayer or JW Player. I think once Amazon creates better controls for private content on their web front-end, this might get easier. I was trying to set it up using a 3rd-party Windows app to configure Cloudfront, and I couldn’t get it to work at all. I was also skeptical how secure this “secure URL” would be, considering that anyone with a little brains who somehow got access to that part of my website could copy the RTMP secure URL the same way they could copy an unsecured RTMP URL. You can get a little fancy in the secure URL by embedding a policy that restricts downloading the file to certain IP addresses or until a certain day, but considering that I couldn’t get even the basic template policies to work, I couldn’t explore the security of the fancier policies.

4) On Cloudfront, the first time someone in each worldwide region requests your content, there will be a slight delay as that content is transferred to a server local to that user. From then it will remain on that local server for 24 hours after the last request. So if I upload the file to the LA server and then my German producer requests it, there may be a very slight delay before it starts to stream for him. However, if he goes back to watch it again 12 hours later, it will already be cached on a local Amazon server and will start streaming immediately.

Use a Droplet to Prep Audio for FCP

Use a Droplet to Prep Audio for FCP

So I’ve had a chance to work for a while on FCP, and I have a few things in mind to post on it. First, though, a very simple trick to help with importing properly configured audio into FCP.

The Issue

FCP, like Avid, prefers uncompressed audio over MP3s, M4As, etc. It’s not that it doesn’t work when you import a MP3 into FCP, it just doesn’t work well. You’ll hear all sorts of clicks and dropouts as you play through the track, so to prevent this you need to convert your audio to WAV or AIF before you import. There is a program called Loader that will convert your audio for you, copy it to a specified folder, and then import it, but it costs $79. The other way you can do two out of those three actions (converting, copying to a specified folder) is to make a Compressor droplet. You still have to import it on your own, but to save $79 I think that’s a pretty good trade-off.

Compressor Droplets

Any compression setting in Compressor can be converted into a standalone application called a Droplet. As you might infer, you can drop things like files onto the Droplet, and it will then process that file into whatever setting the Droplet is programmed with. You can also specify a destination folder for that Droplet and tell it to run silently. Each file you drop onto it will queue up and encode immediately, with the resulting file being placed in your predetermined destination folder.

For me, I have two droplets, one for Mono and one for Stereo (both 24-bit/48k AIF)

Stereo Compression Setting

The image above is for the stereo setting, from which I then make a Droplet in the Settings window:

Make a Droplet

Once you click the Make Droplet button, a dialog will come up asking you where you want to save the Droplet itself, as well as which Destination (defined in the Destinations tab) you’d like that Droplet to send its files to:

Saving a Droplet

Once you click Save, your Droplet will appear wherever you saved it (on my Desktop in my case), and you can proceed to drop files onto it. The first time you run the Droplet a dialog will come up confirming your settings, and you should uncheck the “Show at Launch” checkbox in the lower left corner so that the Droplet runs silently in the future.

Once the compression is done (keeping Batch Manager handy is a good idea to check progress, though most audio takes a very short amount of time), you can import your new files into FCP and cut away.

Screenshot of the Hellboy 2 Avid Project Window

Organizing a Project’s Reels and Versions

This article is a little outdated (sorry!). I need to do a thorough rewrite, but in the meantime know that since 2008 I’ve decided to contradict my writing below and just use R1v## as my reel format. When I hit a milestone I make a new bin, label it as Director’s Cut or Assembly or Preview #2, throw the appropriate sequences in there, and then stash the bin in an easy to find folder in my project. It works pretty well, and I’ve found the fewer characters in your reel names, the easier it is on everyone.

 

The determination of how to organize all of the sequences you’ll create during the course of your project is extremely important, but unfortunately it is also nearly impossible to standardize. I work in features primarily, so while my organizational system is based around reels and suits me just fine, it would need a lot of modification were I to go to episodic or reality TV, commercials, etc. Individual editors also have their preferences for organization based off of how they are accustomed to working, and if you’re an assistant editor your degree of influence over the project’s organization will vary depending on how much the editor wants to delegate this type of organization to you.

With all that in mind, I wanted to set forth a description of the system that I use to keep all my reels in order on a feature. I’ve seen quite a few organizational systems used in different editing rooms, both on films I’ve worked on and ones I’ve just visited, and so far this is the best one I’ve seen. Thanks to editor Sean Albertson and 1st assistant editor Seth Clark for introducing it to me on Rocky Balboa.

Organization By Milestone

One of the most frequent organizational structures I see used by others uses a simple version number attached to the name of the reel. The problem with this is that you get up to these massive version numbers (R1AB_v72, R2AB_v117, etc.), and there’s no common element to link all these versions together in a meaningful way. The system I use has two levels of versioning included in the name of the reel; one is to track a milestone like a director’s cut or a preview screening, and the other is to track the individual versions of a reel created leading up to that milestone. For example:

A sequence name I might use is: R3_DC1_v4, which breaks down like:
R3 = Reel Three
DC1 = Director’s Cut 1 (this is the milestone)
v4 = The 4th version of Reel 3 for Director’s Cut 1.

Using this system, I can not only group my bins and sequences according to the progress of the project, but I can also make sure other departments are working with all of the right reels quickly and easily. Additionally, as it’s often the case that a director/studio/editor wants to go back and look at a previous version, with this system the editor can instantly find and pull up an old version from the project window so long as he knows roughly when the version he is looking for was created.

Screenshot of the Hellboy 2 Avid Project Window

Screenshot of Hellboy 2 project organization. In the project window you can see DC1-DC6 folders. The bins on the right show DC7

Defining Reels

This is global across all organizational systems, so feel free to skip this section if you already know about delineating reels.

Once you’ve got all, or at least most, of your scenes cut, you’ll probably want to create reels out of everything. The general guidelines for creating reels are that your first reel be no longer than 1600 feet from Picture Start to Last Frame of Action (LFOA). All remaining reels should be no more than 2000 feet, though there is a little bit of slack on all of these numbers. With the exception of the beginning of R1 and the end of your credits, no reel should start or end on black. Similarly, don’t cut a reel in the middle of a piece of music. The best reel breaks are between scenes that have no picture or audio transition, so that the break is as seamless as possible. You also want to avoid cutting a reel in the middle of a scene, even if there are no picture or audio transitions, since you may find that the color of your reels varies slightly and a reel break in the middle of a scene could produce a color shift.

Defining Milestones

Ok, so assuming that you decide to use this system, it’s important to know when to create a new milestone. I usually create milestones for the Editor’s Cut (EC), the Director’s Cuts (DC1, DC2, etc.), Previews (PV1, PV2, etc.), and Locks (LOCK1, LOCk2, etc.). And if you’re wondering why I allow for multiple locked cuts, I’d like to know what director you’re working for and if he or she is hiring.

You don’t want to create too many milestones or you risk confusing all of the departments that rely on your organizational system. At the same time, don’t create too few since that will defeat the purpose of organizing your reels this way and make finding old cuts much more difficult. I find that the key to creating a new milestone is to decide whether or not you’ve reached a point in the project that you want to save, including if you’re about to make a very significant change that you want to delineate from previous versions. To rephrase the question, you can also ask yourself whether you’re going to need to create an output of this sequence anytime in the future. For those milestones I mentioned above, clearly you want to have easy access to them since by the time all is said and done you will likely have needed to make more than a few outputs for the studio/director/sound dept/etc.

A final note: If you like, you can simplify the creation of milestones by sticking to a single abbreviation like “DC”.  On Hellboy 2 my milestones went from DC1 – DC7 because everything was happening so quickly and there were constantly new hires coming in for sound, so I felt it was safer to keep the system as simple and predictable as possible. The downside of this is that it’s not as clear to anyone who looks at this project later what each DC number represents, even though I know DC3 was our first preview, and DC6 was our first lock.

Defining Versions

How you elect to version your sequences is something that needs to be figured out between the editor and assistant editor. Some editors I work with prefer to manage their own versions and just rely on me to define the milestones. Others prefer that I manage everything and create new versions nightly or weekly. In the end it doesn’t really matter how you version the sequences so long as the system works for you and, at the end of a milestone, you can pop open your reel bins and see what the final version of that reel is for that particular milestone.

Conclusion

Hopefully this has been a clear explanation of how I like to organize my reels on a feature film. Please let me know how it goes if you decide to try this out, and if anything is unclear or needs further explanation, leave a comment below!

Avid Locators to FCS Chapter Converter

Avid Locators to FCS Chapter Converter

As a supplement to my post on how to convert an Avid locators file to chapter markers for Final Cut Studio, I decided to just make a little web app to do it for you. Paste the contents of your locators file in the top box, click Convert, and then copy/paste the contents of the bottom box into a new text file and import that into DVDSP or Compressor.

If you forgot to change the starting timecode of your sequence Hour 0, you can batch add or subtract a certain # of hours using the control below.

Enjoy!

Organizing an Editorial Department Using Backpack

UPDATE: So my experiment with Backpack came to a premature end. I took another job that came up unexpectedly, so I never got to finish testing out Backpack. I would like to go back to it, though, or try out Basecamp instead. Next time!

Once the craze of Production is over and a temporary period of relative relaxation (known as the Director’s Cut) begins, it becomes time to clean up any messes leftover from Production and truly get the department in order. I’ve decided to use the cleanup period I’m currently in to test drive an organizational web app called Backpack. Made by 37Signals, Backpack is like a ready-made intranet for your office. All you need to do is get people to remember to use it.

I’ve had Backpack up and running for two weeks (out of a 1-month free trial) as of the writing of this post, and it’s been useful enough that I’ll probably start to pay for it when the month expires. For 6 users, which is convenient since there are 6 of us in Editorial right now, it’s $24/month.

Thoughts

I’ll update this as we get further along in our testing, but having an Editorial intranet has long been something I’ve wanted to try out. Since starting to work as an assistant editor, I’ve been conscious of the sheer amount of information I have to remember and then share with the rest of my Editorial team. If I’m the only assistant editor to stay on through both production and post-production, that means I’m the only one who knows all the ins, outs, and irregularities of the film, and bringing other people up to speed takes a lot of time. I’m also a big proponent of open access to information, and I like it when every person in the department can be self-sufficient to a great degree. I don’t want to micromanage a Quicktime output, for example, so I like the other assistant editors on my team to have access to all the same information, email, and contacts that I do, and I find that sharing information freely increases the efficiency with which things get done.

The only barriers stopping me from doing this before were the need to have an in-house server, and that I would be the one who’d have to set it all up and maintain it. Since my main job responsibilities kind of preclude me from having the time to setup and maintain a server, going with another company’s web app is a very appealing option. In addition to Backpack, I also considered Zoho and Google Apps For Your Domain, but in the end the simplicity of Backpack won out.

The only thing so far that has irked me a little is that in its strive for simplicity I’ve found the documentation a little too sparse. For instance, including a Google Map inside a Page turned out to be a piece of cake, but to do that you have to add a Note inside your Page, and nowhere does it say how much HTML/CSS/Javascript you can include. Regardless, here are a few of the highlights of using Backpack.

Journal

The Journal seems to be the least touted feature on the Backpack product page, but it’s where I find all of us spending most of our time. The Journal is akin to a Facebook News Feed. It consists of two boxes, one to list a status update of what you’re currently doing, and another to describe what you have completed. The status update is temporary, and keeps no log of your actions. When you add a completed action, though, that gets appended to the timeline so you can see what people have finished.

As you can see below, it’s useful to for staying informed on what everyone is working on, what tasks they’ve completed, and where to go for lunch.  It’s a pretty simple feature, but it gives us great at-a-glance information on what tasks have been completed and how the day is being spent.

Backpack Journal

Backpack's Journal page (somewhat Photoshopped)

Calendar

On Hellboy 2 our Post Supervisor entered our post schedule into iCal and then shared it through mac.com. That made it easy for all of us (since we were all on Macs) to subscribe to his iCal calendar and stay up to date the instant he made changes to the schedule. I didn’t need to keep track of multiple versions of a calendar that was emailed to me as a PDF, and at the moment I needed to know what was happening I could just open iCal and see the most current schedule reflected right there. For the film I’m on now, I wasn’t willing to shell out the cash for mac.com (now MobileMe), but I still wanted an online calendar that everyone else in the office could subscribe to. For this you can use something like Google Calendar, which I was already using before trying out Backpack, or you can use Backpack’s built-in Calendar.

There are two ways you can use the Backpack Calendar. One is to have Backpack sync its internal calendar with your externally-hosted one (Google Calendar, MobileMe, etc.). The other is to use Backpack as your calendaring software. Since I already had our post schedule on Google Calendar, I just created a Backpack calendar that subscribes to my Google Calendar feed. With this setup, Backpack will sync itself every hour or so to your GCal feed. There is no way to manually sync Backpack, or to change the interval with which it syncs to GCal. Here at the office, I have everyone subscribe their iCal software to my Google feed as well, so that they can refresh iCal at any time, which makes Backpack’s calendar a well-positioned reference, but not the actual source of our online schedule.

Any calendar on Backpack has an iCal address also, so you can subscribe to your Backpack calendar if that’s where you’ve entered all your events and stay up to date in iCal that way. You can also subscribe in iCal to a Backpack calendar that is in turn subscribing to a Google Calendar, but then your iCal is at Backpack’s mercy as to when it updates to reflect new changes.

Backpack's Calendar

Backpack's Calendar

Pages

Last of the features I’ll go over are Pages. This can be like your own little wikipedia. When you create a page, you can set it so that anyone can edit it, and through this they become useful as ever-updating reference for commonly-accessed information like FTP logins and crew contact info.  You can add checkboxes, HTML, images, and attachments to a page. To date, I’ve created a page for:

  • Server logins: Here, everyone has access to all of the FTP, Digidelivery, iDisk, and web site credentials necessary to transfer material to/from Editorial. When we get information on a new server we have to access, anyone can update the server logins page so that it’s easily accessible to others in the future.
  • Delivery specifications: This is where I put all of the Quicktime/tape specs for any department we deliver material to. Right now we only have the music information filled in, but when sound, vfx, and mixing stages start up, we’ll add all their info so that if I ask another assistant editor to make a Quicktime for sound, they can reference what that Quicktime spec is.
  • Basic info: I created a simple page with the address and contact information for everyone in Editorial. When an editor is on the phone and wants to know what our fax number is, they can look here (or shout at me across the hall). I also store our full contact sheet here for quick reference, as well as a condensed version that can be emailed out to new vendors.

Backpack Pages

Backpack Pages (map location altered for privacy reasons)

The Rest

The remaining features of Backpack are Reminders, Writeboards, and Messaging. Reminders are what you’d think: you set a To-Do item and it reminds you by email or SMS when that To-Do item is due.

Writeboards are similar to pages, except with version control. Every time you save a Writeboard, it makes a new version and archives the previous one. You can revert or compare versions at any time, and again, you can set it so that everyone can edit them. I’m trying this feature out for edit notes that need to be addressed, but the way the interface is organized makes the Writeboards a bit buried, especially if you want to include one in a Page. The other thing is that unless you need version control, just working on a Page might be a bit more user-friendly.

Messaging is basically just an internal forum. You can initiate a message to everyone (or certain people), and they can log on and reply. I haven’t found a need for this feature yet in terms of something specific to Editorial, but as a way to get everyone’s Starbucks orders it’s turned out to be pretty handy!

Print a Hierarchical List of Disk Contents

VFX companies often include paper printouts of a hard drive’s contents when they send material to/from Production, the DI, and other vendors. Occasionally, it’s useful for Editorial departments to do the same. Having a printout attached to a hard drive you’re sending out lets your vendor see exactly what’s on the drive without having to plug it in, and can be used to log a drive in or catalog it while it’s offline.

Strangely, there’s a lack of software available to do such a simple task as printing a directory listing. Searching through Google, I found a few applications for OSX that offered this functionality, but either the application wasn’t exaclty what I was looking for, or it added too many flourishes like icons and such.

So, I turned to the Terminal. Using a pattern I found in a forum on Unix.com, I modified the arguments and incorporated it into an Automator workflow (attached). This workflow displays a recursive list of directories (if you want only files or both files and directories see the bottom of this post), and is broken into the following steps:

  1. Asks for the root folder you’d like to make a listing from. This would usually be the root of your external hard drive.
  2. Asks how many levels of recursion you’d like.
    • Example: In one instance I was backing up P2 cards. I only needed to list the folder name of each P2 card I was including, and not the CONTENTS, VIDEO, VOICE, PROXY, ICON, and AUDIO folders that exist as subfolders of a P2 card’s root directory. So using this function with a recursion level of 2, my printout stopped processing directories after 2 levels of hierarchy.
  3. Runs the shell command:
find "$2" -type d -maxdepth $1 ! -name '.*' -print 2>/dev/null|awk '!/\.$/ {for (i=1;i<2  && i != 1 )d=5;printf("%"d"s","|")}print "---"$NF}'  FS='/'
  1. Saves a text file with the listing in the same directory you chose in Step 1, and then opens that file so you can browse and print it.

The output looks like this:

|      |---Warrior Back Up
|      |              |---Dailies by Tape
|      |              |              |---CT001
|      |              |              |---HI-8
|      |              |              |---VT001
|      |              |              |---VT002
|      |              |              |---VT003
|      |              |              |---VT004
|      |              |              |---VT005
|      |              |              |---VT006
|      |              |              |---VT007
|      |              |              |---VT008
|      |              |              |---VT009
|      |              |---P2 Backup
|      |              |        |---MISC
|      |              |        |---PRODUCTION
|      |              |        |---REHEARSAL FIGHTS
|      |              |        |---TESTS

Future Refinements

Suggestions welcome, but things I'm already thinking to add to this workflow later on are:

  1. An option to show files as well as folders. You can change this in the workflow yourself by changing "-type d" to "-type f" for a files-only listing, or remove "-type d" completely for both files and folders.
  2. A dialogue to ask if you want to print the listing automatically

An Introduction to Regular Expressions

Regular Expressions are definitely one techie level up from your traditional tech tip, but they’re definitely worth the time to learn, even if you only learn the basics. A regular expression is very much like a math formula, and you use them when you want to find (and replace) pieces of text using a condition instead of knowing the text in advance.

Conceptual Examples

For example, let’s say you have a file that contains a bunch of phone numbers. And let’s say those phone numbers are all written out as “8005551212”, but you want them to look like “(800) 555-1212.” Using a text editor that supports a regular expression Find & Replace, you could easily reformat all of those phone numbers to include parentheses and a dash, without going row by row to manually change them all. Since you know that your phone number is a string of 10 continous digits, you can tell your text editor to find all instances of 10 numbers in a row, and to insert a ‘(‘ before the first digit, a ‘) ‘ after the third, and a ‘-‘ after the sixth.

A second example is as follows. You have a database full of vfx shot names and shot durations. You also have a sequence full of vfx shots you need to turnover, and every one of them needs a title added to it dictating the shot name and how many frames it is. You can export the information you need from your database, but only as a comma-separated values file (.csv). This would give you results such as:

VFX_001,25
VFX_002,38
VFX_003,119
VFX_004,350
VFX_005,8

To create all those titles, you know that you can use the Autotitler function in Avid Marquee, but the text format it requires is different from CSV, resembling something like this:

VFX_001
25

VFX_002
38

VFX_003
119

VFX_004
350

VFX_005
8

So to reformat your CSV file into the format that Marquee requires, you can tell your text editor to replace every comma in your file with a line break, and to turn every pre-existing line break into a double line break.  You can do this with one regular expression that both replaces the comma and adds a second line break, but I sometimes like to break it up into separate steps to keep things simple.

To demonstrate how to do this find/replace, I’m going to double the line break before I replace the comma. This way I can be sure that I don’t add more line breaks than I need. So the first find and replace would look like this:

Find: \n
Replace: \n\n

In regular expressions, “n” is the notation you can use for line breaks (sometimes also “r” is used either instead of or in conjunction with “n”, but you can google the difference on your own). So what this find/replace does is search for a line break and replace it with two. Then, you can probably guess what to do with the commas:

Find: ,
Replace: \n

This will give you the format you need for the Avid Autotitler.

Lastly, you can also use Regular Expressions in many file renaming utilities (NameMangler is one I use), so if you need to rename a bunch of files in order to conform to a certain pattern, regular expressions can help. One instance where you might use this would be to conform a bunch of irregularly named files in order to put them in sequence for import into an Avid bin.

Regular Expression “Variables”

What the example above is intended to demonstrate is the concept of searching for a pattern of text, rather than knowing what text you’re searching for in advance.  And in order to search for patterns, you must be able to use placeholders to represent certain characters or groups of characters.

This Regular Expression Reference lists the different placeholders you can use when searching text. The ones you’ll use most often are:

  • \d : Finds any numerical character (ie. 0-9)
  • \w: Finds any word, with a word being defined as a group of alphanumeric characters or an underscore, but not including a space
  • \s: Finds any whitespace, including a space, tab, or line break
  • \t: Finds any tab character
  • [ and ] : If you wish to limit the characters you’re searching for, put those characters inside of [ and ]. So for example, [A-Za-z5-8] would find any character from A-Z regardless of uppercase or lowercase, as well as any number between 5 and 8

You will often need to specify how many characters you’re searching for, in which case you’ll need these basic placeholders:

  • ?  : A question mark after a character or character class denotes that you are looking for 0 or 1 instance of that character
  • *  :  An asterisk denotes you are looking for 0 or more of that character
  • +  : A plus sign denotes you are looking for 1 or more of that character
  • { and }  : These brackets allow you to say exactly how many characters you want to match. For example, “\d{2}” tells the program you’re searching for a string of exactly two digits.  “\d{2,8}” tells the program you’re searching for between 2 and 8 digits, and “\d{2,}” specifies that you’re searching for at least 2 digits.

And lastly, you’ve seen the backslash (\) used a lot here, and that’s worth explaining. In regular expressions, the backslash functions as what’s called an escape character. The rules of regular expressions are a bit complex, and many characters you may want to search for have functional meanings, like the fact that an asterisk (*) tells the program to match 0 or more characters. If you want to search for an asterisk, though, you may need to escape it. And you do that by putting a backslash before the asterisk, like so: \* .  By using the backslash, you are either telling the program to ignore the special meaning that a particular character has, or to match a character that is not easily defined (like \t, which represents a tab character).

Back References

The last concept I want to explain can be tricky to get your head around while you’re still digesting everything else, but it’s a very useful thing to know, and is called a back reference. Let’s take the timecode example below… In this situation, you have a bunch of timecodes without colons (:) separating the hours, minutes, seconds, and frames (ie. 01020304). You want to insert the colons, but you need a way to tell the program not to throw out the digits that make up the timecodes when replacing the timecode text. So to do that, you have to save those digits during the Find part of the process for use during the Replace part. You do this by enclosing the text you want to save in parentheses, as so: (\d{2})

Then, in your Replace expression, you can tell the program to insert the text it’s saved by including $1, $2, $3, and $4. The first parentheses in your Find expression are referenced by $1, the second by $2, and so on. And when replacing the timecodes, if I put a set of parentheses around every 2 digits, that will allow me to then insert colons between those pairs of digits, thus giving me properly formatted timecode in the form of 01:02:03:04.

Examples

The easiest way, I think, to grasp how you use all these placeholders is to show you some examples, some of which come from this Regular Expressions site.

Email Address:

This will match most email addresses,

\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,4}\b

and is broken down like this:

  1. \b matches a word boundary (most likely a space)
  2. [A-Za-z0-9._%+-]+ matches any alphanumeric character, regardless of case, as well as the punctuation also enclosed within the brackets. The + sign at the end states that you are looking for 1 or more characters that match this pattern, since most email addresses are more than one character long.
  3. @ simply matches the @ sign in an email address
  4. [A-Za-z0-9.-]+ will match the server name in your email address (ie. it will match the “gmail” in “me@gmail.com”)
  5. \. will match the dot between your server name and your top-level domain (ie. it will match the “.” in “me@gmail.com”)
  6. [A-Za-z]{2,4} will match the .com, .org, .net, .info, or whatever you happen to have, by matching 2-4 alphabetical characters
  7. \b again matches a word boundary, presumably a space or line break

Timecode:

This will match timecode, which I’ve used in the past to reformat a subtitle file from an Excel-exported CSV into a DVD Studio Pro formatted .stl file. Below is my source file, which as you see is missing the “:” in all of the timecodes.  The .stl file requirements ask that there be a space on eiher side of the commas separating the TC and subtitle text, which I did in Excel by concatenating several cells into one column with the appropriate comma spacing.

Source CSV File:
01052021 , 01052328 , …but now I wonder if it is just the fear talking.
01052419 , 01052800 , I'd like to say I'm the son of a famous person.
01052812 , 01053106 , Or at least someone who is politically affiliated.
01053128 , 01053406 , But that is not the truth.
01053427 , 01053619 , So the only reason I can  think of…
01053800 , 01053922 , …is money.
01055425 , 01055914 , -Be reasonable. | -Don't you understand we make the rules here?
01060003 , 01060209 , We will give you exactly what you want.
01060210 , 01060517 , -Make sure of it. | -It has been 19 days!
01060518 , 01060625 , That doesn't matter anymore.
01060626 , 01060815 , It does matter.
01061309 , 01061603 , I already gave you till noon.
01062605 , 01062800 , Don't do this.
01063622 , 01063729 , Wait.

To find and replace the timecodes, I would use these patterns:

Find: (\d{2})(\d{2})(\d{2})(\d{2})
Replace: $1:$2:$3:$4

Breakdown:

  1. (\d{2}) searches for strings of 2 digits, and the fact that the \d{2} is within parentheses means that the program will save the two digits it finds so that I can reinsert them when replacing the text. Since I know my timecode is 8 digits long, I put four of these statements in a row so that I can keep the hours, minutes, seconds, and frames separate.
  2. $1:$2:$3:$4 replaces every 8-digit timecode string the program finds with the first two digits it saved from the parentheses, followed by a colon, followed by the second pair of digits, then a colon, etc. This is a back reference, as mentioned above.

Conclusion

I’ll add to this article as new examples and uses arise, but hopefully this and Google will get you started on figuring out all the different ways you can use Regular Expressions. If you’re confused about when to use them, just stop yourself when you find that you’re in a position of having to make a bunch of tedious edits to a text file. It may be that you can save yourself a lot of time and typing by using a Regular Expression Find/Replace.

Additional References:

PHP: Regular Expression Details

Hellboy 2 Turnover Workflow

Introduction

Turning over reels from HD material in your Avid to sound, music, and marketing departments is a process that requires a lot of rendering time no matter which way you do it. At the outset, your two options for getting your Avid material into Quicktime format are to either export the reel directly to QT, or to play out in real-time to another computer running the capture tool in FCP. Below is a table of all the turnover specs for each department, which I think are pretty standard requirements, followed by a discussion for what we tried on Hellboy 2 and how we ultimately chose to accomplish all of these turnovers.

[TABLE=2]

Exporting a QT vs. Playing Out to FCP (for SD Turnovers)

As you can see in the table above, the majority of our turnovers were Quicktime-based. If the Marketing Department hadn’t required an unmasked image, I probably would have used my sound turnover QTs to make their DVD, too. And if I had more time to experiment, I probably would’ve devised a way to export ‘blank’ QTs without a matte and then apply the matte in Compressor, which I actually did do on a few isolated occasions but never ended up incorporating into the workflow. But anyway, moving on…

As I mentioned above, there is no part of making an SD turnover from your HD material that is quick and painless, unless you are going to something with a 29.97 frame rate. The problems are basically these:

  1. Exporting an average-length reel (~20 minutes) from 1920×1080 in the Avid to a 720×486 (or similar-sized) QT takes an average of 1.5 hours per reel.
  2. Playing out via an SD output on the Adrenaline to another computer running FCP means that you’re going out of the Avid at 29.97fps, without giving FCP a way to know where your A-frame is and so reliably bring you back to 23.976. If all you need is 29.97 SD, then the biggest drawback to this is just the quality of whichever type of video output you use (SDI, Composite, S-Video, etc.)
  3. If you play out via HD-SDI to an FCP station capturing in HD, you still face the task of compressing the HD QT that FCP creates down to SD. If you figure 20 minutes to play out, plus roughly an hour to trim the QT and compress to SD, you’re looking at about the same amount of time as an export.

A black box hardware solution that takes HD-SDI and outputs a QT of the specified size in real-time without adjusting frame rate would be great. There must be something like this in existence, but I haven’t heard about it yet and haven’t seen anything in my searches that sounds like it does what I want. So since I couldn’t find a box like this, and since the tedious process of rendering, playing out, and compressing down to SD was roughly equivalent in time to exporting a reel directly to QT, I chose to go the direct export route. Ultimately, that meant a simpler turnover process for us in Editorial, as well as higher quality video for the departments who would be receiving the turnovers.

An Exception:

When we started our turnovers, I did go the playout route for our Composer and Music Editor. Their specs had originally asked for 320×240 QTs @ 29.97, but when I sent them a 720×486 QT just to test out, they decided they liked the higher quality files (who wouldn’t?) and changed their spec. So I played out through a Canopus box to FCP, trimmed the QTs and sent them on their way until one day we decided to test exporting a 29.97 from our 23.976 Avid project. The frame rate test worked (going from 24.000 -> 29.97 still doesn’t, I believe), but at the same time the Music Editor was testing the new QT he was noticing how much higher quality the exported video was. And so the workflow changed again…

Exporting Reusable Quicktimes Efficiently

One of the biggest tricks I developed on HB2 was to create Quicktimes that I could reuse. Since the biggest outlay of time required in turning over is exporting HD material to an SD Quicktime file (AAFs, guide tracks, and EDLs require much less time, and can all be done while the QTs are exporting with time to spare), the more work you can divert to Compressor instead of Avid, the better. We ended up exporting “blank” Quicktimes, which contained almost all of the visual burn-ins required for all of our turnovers except for individual initials. We also exported these Quicktimes without any audio, since it takes time and rendering to pan audio tracks in the Avid (and problems multiply if you have any Time Compression effects), and since you’re limited to 2-track stereo on any QT export from Avid.

This is what our “blank” Quicktimes looked like after they were exported mute from the Avid. They have a 1.85 matte covering the key numbers, the requisite “Property Of”, the version of the reel, and the date.

blank_qt_burnin1

After these Quicktimes finished exporting, we would then take them into Quicktime Pro and add the audio guide tracks we had exported on a different Avid while the QT was rendering out. Each guide track was added as a separate audio track to the QT, which in the end gave us a QT with three audio tracks (DX and FX as mono AIFFs, and MX in stereo). We would then go in and set the channel of each audio track to either Left or Right depending on what we needed (this is changeable at any time, including after running through Compressor), and do a Save as Self-Contained Movie (~1 minute per file).

Once the audio was added and panned appropriately, we would then take it into Compressor for creating individualized Quicktimes. Our basic Compressor settings simply kept the video codec as it was (Motion JPEG A, Medium Quality), and set the audio as Pass-through so as to keep our discrete three-track audio. We then added a text overlay, saved the setting and started the render. Each reel takes approximately 10 minutes to render (keep in mind that results vary by codec), and produces a new, basically identical QT file that looks like this:

personalized_qt_burnin

HD Turnovers

There’s a lot less to talk about when making HD Quicktimes from an HD Avid project. Given the appropriate equipment, which I probably could’ve gotten on HB2 but never did, I think playing out HD-SDI would be the fastest way to go. Video quality is obviously not an issue when playing out HD-SDI, and there are no tricky frame rate issues to tackle. We generally only had to export a single set of HD Quicktimes, just for our mixing stage, so in that case we included the personalization in the Avid title we applied to the reels before export, and eliminated the need to go to Compressor afterwards. As the show neared completion we had a few HD-capable sound and foley stages requiring turnovers, so only then did we use the “blank” QT export -> Compressor personalization process. But again, given an AJA or Kona card, I’d probably would’ve just played each one out.

Summary

It is a lot of steps to create template Quicktimes, but what this workflow gave us was the flexibility to make multiple, customized copies of our reels quickly, easily, and in batch. I can’t count the number of times we were asked to create additional copies with slightly varying specs or burn-ins, and being able to do that without re-exporting from Avid was a life saver. Additionally, Compressor can be happily processing copies in the background while you continue to work in Avid. It may slow down the Compressor renders a little, but as long as you’re not doing anything too processor-intensive in Avid, you should have no problem with running both programs concurrently.

Timewise, the following are pretty accurate estimates, although it will clearly take some more time when you first set up your Compressor workflows or go through the Quicktime Pro process.

  • Prep burn-ins and matte in Avid – 2 minutes / reel
  • Export from Avid – 1.5 hours / 20-min reel  ** (This will change as processor speeds increase. Also, some people have had luck with creating Reference Quicktimes, but I haven’t.)
  • Export Guide Tracks – 5 minutes / reel (assuming you’ve split out your tracks before starting to turnover)
  • Add audio tracks to exported QT file in Quicktime Pro – 3 minutes / reel including time to save as Self-Contained Movie
  • Add individual burn-in using Compressor – 10 minutes / reel

Bonus

Aside from just sticking someone’s name on a QT, here are a few other things you can do with QTPro and Compressor:

  • Make a DVD of the whole feature
    • In QTPro, mark an IN/OUT on each of your ‘blank’ reels at the FFOA and LFOA, then copy and paste in reel order into a new movie. Verify your audio panning and reel breaks (wouldn’t want the reels in the wrong order!), then just Save as a reference movie. Take your reference movie into Compressor, make a new workflow (you can resize/add a matte or text as needed), and render to MPEG-2 and AC3 and burn to DVD in the software of your choice. Voila, 45 minutes of rendering to make the MPEG-2, ~10 minutes for the AC3, and you’ve just made a high quality DVD in less time than it would’ve taken to play it out analog.
  • Save time by rendering two halves of a reel simultaneously (Verify your results. 90% of the time this works flawlessly, 10% of the time there are inexplicable problems at the split point after Compressing)
    • If you have multiple Avids available and are short on time, you can render half of a reel on one Avid and half on the other. Make sure you render with the Use Marks option checked, and that your OUT mark on Avid #1 is adjacent to the IN mark on Avid #2. Once you have each half reel as a QT, copy/paste one half into the other, add your audio, and continue with the process by doing a Save as Self-Contained movie before bringing into Compressor. If you don’t have audio, you can just save as a reference movie. Always verify your results after running through Compressor, I’ve run into unexplained repeated frames at the splice point in my output QTs when doing this and have not been able to reliably recreate the conditions that cause them. But if you’re in a rush, it can be worth the risk.
  • Letterbox
    • I usually output my 720×486 turnover QTs as anamorphic, in part because it’s faster than making everything letterbox, and so far no one’s cared. On my next show I’ll use 864×486, since that’s the proper aspect ratio, and I don’t think I need to take NTSC client monitors into account anymore. Anyway, if you need to do a letterboxed 720×486 QT from an anamorphic original, Compressor can do this for you during your export. Just add the Letterbox option to your workflow and select either Matte or Resize accordingly.

File Delivery Methods

Introduction

For anyone who hasn’t worked in London’s Soho district, it is a dream come true in terms of having all of your post-production departments in one place. We were fortunate enough to be based there after we wrapped Hellboy 2 in Budapest, and it meant that our sound department was two doors down, our VFX vendor was at the end of the street, our mixing stages were right next door, our DI house was a 10-minute walk, and the meeting point in case of a fire was the pub across the street (and they served Pacifico, in case I got nostalgic for something Mexican). What this meant for us in terms of our turnover process was that delivering the turnover material was quick and painless, even if creating it wasn’t.

So here’s the rundown on how we delivered files to the people who needed them, and my preferences for delivery mediums:

  • Firewire Hard Drive
    • This is pretty straightforward, and by far the easiest way to get your files from Editorial to whoever needs them. I happen not to be a fan of LaCie drives due to the unreliability of their FireWire ports, but they’re cheap, replaceable, and their rugged versions are available with a triple interface (USB, FW, FW800). If you go cheap and use LaCie, just make sure you buy them from a vendor who’s willing to take them back and provide a replacement when the FW port fails. This happened to us twice (and I’ve had this experience on previous shows as well).
  • DVD
    • Do not burn Quicktimes to DVD. I have no information whatsoever on why this doesn’t work, but it doesn’t and you’ll waste your time trying to make it work reliably. Everything will appear to be ok when you burn the disc, but trying to copy the QT off of it fails. For security reasons, I also dislike DVDs because they rely on your recipients to properly dispose of or store them after they copy the files off. So because of these two limitations, I’ve abandoned this method entirely in favor of sending a courier on a return route with a hard drive.
  • USB Flash Drive
    • These work well for deliveries of small quantities of files. Make sure you plug your USB drive directly into the computer and not into a monitor or keyboard USB port (those are usually USB1 and are very slow). If you have a large quantity of small files, I recommend that you zip them up before copying to the flash drive, and then unzip after copying the zip file from it. Just due to the nature of how it works to copy files to/from a flash drive, it is faster to copy, for example, one 2GB Quicktime than it is to copy a thousand small text files. Just remember to clear off your flash drive when the file’s been delivered. These things have a habit of being left out in the open for anyone to swipe.
  • Electronic Delivery Methods
    • I’m a big proponent of delivering files electronically. It saves me time the time of copying to a drive and arranging delivery, and if the transfer process is well set-up, you can transfer direct from your network to the recipients’. If your transfer process involves you copying to a 3rd-party server, and then your recipient downloading from the same 3rd-party server, it might save more time to go the physical route with a FW hard drive, depending on your circumstances and geographic location.
    • FTP
      • Simple, easy to use (for some), and fast, FTP is a great option for non-confidential files. While the likelihood of anyone eavesdropping on your FTP session is slim to none, FTP is technically insecure. It transmits your passwords in plain text, and because of that if someone were to listen in they could, in theory, gain access to your account. But if that’s not a concern, then FTP has everything going for it, including the fact that anyone can set up an FTP server, and this frees you from using some poorly-designed corporate solution.
      • Tip: The concept of FTP seems to constantly elude non-techie users. If the recipient clearly has never heard the term “FTP client” before, don’t try to have them download Filezilla or Cyberduck, but instead first try to give them a URL a browser can read. I usually include a URL like this in any email to producers when using FTP:
        • ftp://username:password@server.com/folder
      • This is a standard FTP link that will work in any browser. If you are on a MediaTemple (gs) server, like I am, then these URLs will not work in Safari or IE, but will still work in Firefox, and look like this:
        • ftp://username@server.com:password@server.com/folder
    • Protected Folders on Websites
      • This has the same security limitations as FTP, in that the transmission of your password to the protected folder is insecure, on the off-chance that anyone is bothering to listen in. But it does allow you to give a standard http:// URL to your recipients, along with an easy-to-use username/password dialog box. I use this method as my last resort, and always remove the files I’m transferring as soon as I know that the recipients have them.
    • Digidelivery
      • Digidelivery is my favorite 3rd-party application for securely transferring files. It’s designed for use with ProTools sessions, but it will transfer anything and everything you give it. The client software is free, but you have to know someone with a server (or own one yourself). If the option is there to use Digidelivery, use it.
    • Signiant / Aspera
      • We used Signiant, and occasionally Aspera, to transfer files back to Universal on HB2. Our experience was mixed, there are things I like about how it’s set up, and things that clearly need some work. Universal has an excellent Digital Media Delivery team, though, that is working every day to improve the software, and I look forward to trying it out again some day. I won’t publish all the ins and outs of my experience using Signiant here, but I’d be happy to discuss it by e-mail if someone is about to start a show that will use it.
    • SmartJog
      • We used SmartJog very briefly on Rambo, actually. It’s a service for secure file transfers worldwide, but as far as I know it does not allow you to transfer to a client not on the SmartJog network, so it’s unlikely that your Editorial office will be set up with it. I think it’s mostly for studios and facilities, so if you’re on a show that’s using it, you will most likely have to send a drive to whatever facility you’re working with each time you want to receive a transfer.
    • Sohonet
      • Sohonet is an ISP, not a file transfer service, but its network is phenomenal, and if a significant number of your departments and/or vendors are on it, it can be fantastic. Basically, everyone who’s on Sohonet’s private fiber optic network can communicate with each other (i.e. FTP or Digideliver files) at speeds of around 80 Mbps (not to be confused with its equivalent of 10MBps). If you’re FTP-ing to someone outside of the Sohonet network, then your speed is just dependent on what kind of ‘external’ access you pay for.