Archive for: April, 2009

Upres a Sequence Multiple Times

This explanation comes by way of Ian Differ, and is useful if you’re cutting on Avid in HD and need to upres an updated version of a sequence you’ve already upres-ed once. For example, if you are working in DNxHD 36 and want to preview a feature film in DNxHD 115, make changes to it and then preview it again, here’s a deceptively simple way to do that.

Note: If you’re on a Unity, I’ve found it helpful to unmount all of my workspaces except for a dedicated one or two that I create specifically to hold higher-res media. This helps you verify that all of the clips in your sequence are linked to the higher-res media and aren’t still pointing at lower-res material.

Doing Your First Upres

Decompose settings

First up-res Decompose settings

  1. Decompose: The first time you want to prepare a sequence for upresing, start by copying the sequence into another bin, selecting it and then choosing Decompose from the Clip menu. (NB: I usually strip off all the Audio tracks before I do this.) Since this is the first time you’re doing this upres, you should have some handles set so that you can change your sequence later and be able to use most of the higher-resolution media you’re about to capture (reusing media being the point of this article!). I’ve selected ‘Captured clips only’ in the screen shot to the right, since Quicktimes are so easily re-imported in their entirety that I probably wouldn’t bother with decomposing them at all.
  2. Capture: Once you’ve got your decomposed sequence, open the Capture tool, select your intended higher resolution (DNxHD 115 or higher), and Batch Capture all of the decomposed clips in your new resolution.
  3. Put it all back together: Once your batch capture is complete, all of the clips in your sequence that originated from tape should be back online. If you’re on a Unity, you can now remount your other volumes, and watch as your Quicktime-based clips come back online as well. If you stripped out audio prior to decomposing, take your original sequence, splice your audio tracks into your decomposed sequence, and you should be good to go.

Doing Your Second Upres

Second up-res Decompose settings

Second up-res Decompose settings

Okay, so now you’re at the point where you’ve probably screened your first preview with high-res media, made some changes as a result, and are preparing to do a second preview screening. You know that you have all this high-res media online, and you’d prefer to reuse as much of it as possible, so now it’s time to go through the process again. If you’re on a Unity, unmount every workspace except the ones containing your higher-resolution media. Then,

  1. Decompose with 0 frame handles: Since you already have your high-res decomposed media online, this time when you decompose you want to make sure to reuse as much of it as possible. You do this by Decomposing with 0 frame handles. If you were to decompose with 48 frame handles again, you would find that even clips that had only been slid by one frame would not relink, since Avid needs to relink the whole clip including handles, even if the range of the clip that exists in the cut is still within the handles of the originally-decomposed media. So by specifying 0 frame handles on your decompose, minor edits will still relink.
  2. Capture remaining clips: New shots, extensively trimmed clips, or basically whatever does not relink will need to be re-captured, but this should be a far smaller task than it was the first time around. When finished, bring your other workspaces online again, splice in your audio, and you should be good to go.

N.B.: In your Timeline’s Fast Menu, you can set your Clip Text to include Clip Resolutions. This is a good way to double-check that your media is at 115 or whatever your higher resolution setting is supposed to be.

An Introduction to Regular Expressions

Regular Expressions are definitely one techie level up from your traditional tech tip, but they’re definitely worth the time to learn, even if you only learn the basics. A regular expression is very much like a math formula, and you use them when you want to find (and replace) pieces of text using a condition instead of knowing the text in advance.

Conceptual Examples

For example, let’s say you have a file that contains a bunch of phone numbers. And let’s say those phone numbers are all written out as “8005551212”, but you want them to look like “(800) 555-1212.” Using a text editor that supports a regular expression Find & Replace, you could easily reformat all of those phone numbers to include parentheses and a dash, without going row by row to manually change them all. Since you know that your phone number is a string of 10 continous digits, you can tell your text editor to find all instances of 10 numbers in a row, and to insert a ‘(‘ before the first digit, a ‘) ‘ after the third, and a ‘-‘ after the sixth.

A second example is as follows. You have a database full of vfx shot names and shot durations. You also have a sequence full of vfx shots you need to turnover, and every one of them needs a title added to it dictating the shot name and how many frames it is. You can export the information you need from your database, but only as a comma-separated values file (.csv). This would give you results such as:

VFX_001,25
VFX_002,38
VFX_003,119
VFX_004,350
VFX_005,8

To create all those titles, you know that you can use the Autotitler function in Avid Marquee, but the text format it requires is different from CSV, resembling something like this:

VFX_001
25

VFX_002
38

VFX_003
119

VFX_004
350

VFX_005
8

So to reformat your CSV file into the format that Marquee requires, you can tell your text editor to replace every comma in your file with a line break, and to turn every pre-existing line break into a double line break.  You can do this with one regular expression that both replaces the comma and adds a second line break, but I sometimes like to break it up into separate steps to keep things simple.

To demonstrate how to do this find/replace, I’m going to double the line break before I replace the comma. This way I can be sure that I don’t add more line breaks than I need. So the first find and replace would look like this:

Find: \n
Replace: \n\n

In regular expressions, “n” is the notation you can use for line breaks (sometimes also “r” is used either instead of or in conjunction with “n”, but you can google the difference on your own). So what this find/replace does is search for a line break and replace it with two. Then, you can probably guess what to do with the commas:

Find: ,
Replace: \n

This will give you the format you need for the Avid Autotitler.

Lastly, you can also use Regular Expressions in many file renaming utilities (NameMangler is one I use), so if you need to rename a bunch of files in order to conform to a certain pattern, regular expressions can help. One instance where you might use this would be to conform a bunch of irregularly named files in order to put them in sequence for import into an Avid bin.

Regular Expression “Variables”

What the example above is intended to demonstrate is the concept of searching for a pattern of text, rather than knowing what text you’re searching for in advance.  And in order to search for patterns, you must be able to use placeholders to represent certain characters or groups of characters.

This Regular Expression Reference lists the different placeholders you can use when searching text. The ones you’ll use most often are:

  • \d : Finds any numerical character (ie. 0-9)
  • \w: Finds any word, with a word being defined as a group of alphanumeric characters or an underscore, but not including a space
  • \s: Finds any whitespace, including a space, tab, or line break
  • \t: Finds any tab character
  • [ and ] : If you wish to limit the characters you’re searching for, put those characters inside of [ and ]. So for example, [A-Za-z5-8] would find any character from A-Z regardless of uppercase or lowercase, as well as any number between 5 and 8

You will often need to specify how many characters you’re searching for, in which case you’ll need these basic placeholders:

  • ?  : A question mark after a character or character class denotes that you are looking for 0 or 1 instance of that character
  • *  :  An asterisk denotes you are looking for 0 or more of that character
  • +  : A plus sign denotes you are looking for 1 or more of that character
  • { and }  : These brackets allow you to say exactly how many characters you want to match. For example, “\d{2}” tells the program you’re searching for a string of exactly two digits.  “\d{2,8}” tells the program you’re searching for between 2 and 8 digits, and “\d{2,}” specifies that you’re searching for at least 2 digits.

And lastly, you’ve seen the backslash (\) used a lot here, and that’s worth explaining. In regular expressions, the backslash functions as what’s called an escape character. The rules of regular expressions are a bit complex, and many characters you may want to search for have functional meanings, like the fact that an asterisk (*) tells the program to match 0 or more characters. If you want to search for an asterisk, though, you may need to escape it. And you do that by putting a backslash before the asterisk, like so: \* .  By using the backslash, you are either telling the program to ignore the special meaning that a particular character has, or to match a character that is not easily defined (like \t, which represents a tab character).

Back References

The last concept I want to explain can be tricky to get your head around while you’re still digesting everything else, but it’s a very useful thing to know, and is called a back reference. Let’s take the timecode example below… In this situation, you have a bunch of timecodes without colons (:) separating the hours, minutes, seconds, and frames (ie. 01020304). You want to insert the colons, but you need a way to tell the program not to throw out the digits that make up the timecodes when replacing the timecode text. So to do that, you have to save those digits during the Find part of the process for use during the Replace part. You do this by enclosing the text you want to save in parentheses, as so: (\d{2})

Then, in your Replace expression, you can tell the program to insert the text it’s saved by including $1, $2, $3, and $4. The first parentheses in your Find expression are referenced by $1, the second by $2, and so on. And when replacing the timecodes, if I put a set of parentheses around every 2 digits, that will allow me to then insert colons between those pairs of digits, thus giving me properly formatted timecode in the form of 01:02:03:04.

Examples

The easiest way, I think, to grasp how you use all these placeholders is to show you some examples, some of which come from this Regular Expressions site.

Email Address:

This will match most email addresses,

\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,4}\b

and is broken down like this:

  1. \b matches a word boundary (most likely a space)
  2. [A-Za-z0-9._%+-]+ matches any alphanumeric character, regardless of case, as well as the punctuation also enclosed within the brackets. The + sign at the end states that you are looking for 1 or more characters that match this pattern, since most email addresses are more than one character long.
  3. @ simply matches the @ sign in an email address
  4. [A-Za-z0-9.-]+ will match the server name in your email address (ie. it will match the “gmail” in “me@gmail.com”)
  5. \. will match the dot between your server name and your top-level domain (ie. it will match the “.” in “me@gmail.com”)
  6. [A-Za-z]{2,4} will match the .com, .org, .net, .info, or whatever you happen to have, by matching 2-4 alphabetical characters
  7. \b again matches a word boundary, presumably a space or line break

Timecode:

This will match timecode, which I’ve used in the past to reformat a subtitle file from an Excel-exported CSV into a DVD Studio Pro formatted .stl file. Below is my source file, which as you see is missing the “:” in all of the timecodes.  The .stl file requirements ask that there be a space on eiher side of the commas separating the TC and subtitle text, which I did in Excel by concatenating several cells into one column with the appropriate comma spacing.

Source CSV File:
01052021 , 01052328 , …but now I wonder if it is just the fear talking.
01052419 , 01052800 , I'd like to say I'm the son of a famous person.
01052812 , 01053106 , Or at least someone who is politically affiliated.
01053128 , 01053406 , But that is not the truth.
01053427 , 01053619 , So the only reason I can  think of…
01053800 , 01053922 , …is money.
01055425 , 01055914 , -Be reasonable. | -Don't you understand we make the rules here?
01060003 , 01060209 , We will give you exactly what you want.
01060210 , 01060517 , -Make sure of it. | -It has been 19 days!
01060518 , 01060625 , That doesn't matter anymore.
01060626 , 01060815 , It does matter.
01061309 , 01061603 , I already gave you till noon.
01062605 , 01062800 , Don't do this.
01063622 , 01063729 , Wait.

To find and replace the timecodes, I would use these patterns:

Find: (\d{2})(\d{2})(\d{2})(\d{2})
Replace: $1:$2:$3:$4

Breakdown:

  1. (\d{2}) searches for strings of 2 digits, and the fact that the \d{2} is within parentheses means that the program will save the two digits it finds so that I can reinsert them when replacing the text. Since I know my timecode is 8 digits long, I put four of these statements in a row so that I can keep the hours, minutes, seconds, and frames separate.
  2. $1:$2:$3:$4 replaces every 8-digit timecode string the program finds with the first two digits it saved from the parentheses, followed by a colon, followed by the second pair of digits, then a colon, etc. This is a back reference, as mentioned above.

Conclusion

I’ll add to this article as new examples and uses arise, but hopefully this and Google will get you started on figuring out all the different ways you can use Regular Expressions. If you’re confused about when to use them, just stop yourself when you find that you’re in a position of having to make a bunch of tedious edits to a text file. It may be that you can save yourself a lot of time and typing by using a Regular Expression Find/Replace.

Additional References:

PHP: Regular Expression Details

Add Virtual Cuts to a Clip Based on the Tracks Beneath

Ever needed to add cuts to a clip at the edit points of each shot inside that clip? Yeah, me too. I’m sure this has uses outside of the feature film world, but since features are what I know, I’ll explain this using an example from doing a DI.

Imagine that your DI house has just finished assembling a reel, and they send you a check tape. You ingest the check tape into your Avid, and cut it into your reel’s sequence on the top-most video layer. All is well and good, except that you have one, unbroken 20-minute clip on your timeline. Sure, you’ve got the video tracks beneath it with your dailies, but you’d really like to add edits to that 20-minute clip where each shot starts and ends.

Fortunately, this is pretty easy to do, but Avid has hidden it away inside of the Pan & Scan effect (as of MCA 3.1.2). Here’s how it goes:

  1. Load up the sequence that corresponds to your unbroken clip. For features, this would be loading up the Reel 2 sequence that your Reel 2 DI check tape is supposed to match, for example.
  2. Add a new video track, and cut in the clip you’d like to slice up so that it is in sync with the video tracks underneath
  3. Add a Pan & Scan effect to the clip you just cut in
  4. Make sure that all your track selectors are active
  5. Go to the settings for the Pan & Scan effect, and under the Action menu, click Subdivide. This will add virtual cuts in your clip wherever there is a cut in the active video layers beneath it.
  6. Using the segment tool, select all of the segments on your newly-sliced clip, and click the Remove Effect button. This will remove the Pan & Scan effect from that track, leaving you with one clip and a bunch of matched-frame edits at the boundaries of each shot.

N.B. I referenced this briefly in Step 5, but I’ll explain it a bit more. When you click the Subdivide button, Avid will insert edits corresponding to whichever tracks are active and have cuts beneath the track with the Pan & Scan effect. So if you want to make your virtual edits based on V1, V2, and V4, but not V3, just don’t have the V3 track selector active when you click Subdivide.

Once you’re finished you should have a timeline looking somewhat like this:

A sample timeline after using the Subdivide effect on the top-most video layer

A sample timeline after using the Subdivide effect on the top-most video layer

Importing 24fps Quicktimes Into 23.976 Projects

Here’s a neat little console command to tell Avid to ignore the frame rate of any imported Quicktime. You would need this command if, for example, you’re cutting in 23.976 but your vfx house insists on delivering 24.000 fps QTs. This would also work in the reverse, if someone delivers you 23.976fps Quicktimes and you need them to be 24.000fps.

Open the console and type in this command:

ignoreqtrate true

To turn it off, just substitute ‘false’ for ‘true’.

This works particularly well in the 24/23.976 situation since there are actually the same number of frames in both types of Quicktimes, they just play at slightly different speeds. So by telling Avid to ignore the frame rate of the Quicktime, you can prevent Avid from trying to interpolate frames to convert 23.98 to 24, since it shouldn’t be doing that anyway.