Any intelligent masking tools?
by Tatha
on
Feb 24, 2004 at 3:39:52 pm
Hi,
I have 2 questions:
1) I have Matt's older VHS series on Commotion. In which tape can I find a description of applying motion tracking to spline verices, as was mentioned by someone in the post concerning roto-ing a hat out?
2) I'd also like to ask a broader but related question. Are there still no softwares/tools to partially automate hand roto-ing using some kind of built-in intelligence? E.g. in many cases it may not be possible to key at all because of a busy background but it may be possible to motion-track a few important points on the subject. If I could then use the trackers on individual spline vertices, then a large part of the mask animation could be automated, leaving only fine-tuning to be done. If not in Commotion, can some other software do this - e.g. Combustion?
Similarly, there could be a software which makes rotosplines "snap" intelligently to the edges of the subject using local contrast once you draw a loose approx garbage mask around the subject (I believe Automasker tried to do this when it was available, but wasn't very good; Motionkey).
Or, better still, some kind of intelligent pattern recognition based software could try to identify objects "semantically" and extract them (hordes of research papers on Univ sites claiming they have successfully done this using human vision models - humans can just look at a picture as a whole and know where the outlines of the subject are). Or, something like what matrix metering does in cameras by cataloguing thousands of typical scenes and then guessing a correct format when applying this database to real life data.
Any way to automate roto-work would make so much more footage eligible for extraction of subject / actions. Any pointers anyone?
Re: Any intelligent masking tools? by J Bills on Feb 24, 2004 at 8:52:48 pm
Not sure what tape it's on, but on my DVD series there is a lesson entitled "tracking rotosplining." That's what you want.
But it's simple to understand. Roto an image - just one frame. Then switch to the tracker, and track the points you would like to use (in theory one for each CV). So say you roto'd a box - well, you'd have a 4 point spline and 4 tracking points.
Then in the roto palette, select the one point on your spline you would like to add the tracking data to, and click apply tracker. Then just make sure you select the option for "selected points only." Apply your tracking data (x and y).
This is all off the top of my head, so don't quote me - but it's something like that. You just have to make sure your image will return usable tracking data - which isn't "normal" by any stretch of the imagination.
Other programs have this ability. In Shake, there is a way to modify a script so that you apply tracking data to an individual point (it's more for advanced users). That has limited functionality.
In flame, it's easy to track an individual point, and you can use the "Tracer" function to automatically track and key an object, which would be similar to what you are talking about. This is usable but I've had very limited success with it - it's more meant for tough bluescreens.
Another similar program is Mokey, which sets up an automated "roto" that in effect is really more of a roto/key hybrid that uses tracking data. It's probably the closest thing to what you are after. But it's incredibly slow, computer intensive, and I have yet to have success with it on a real job. Although it will do the tutorials just fine! It's also quite expensive.
2D3 is developing a program called pixeldust that is supposed to be similar to mokey. If they would ever bring it out!!
After Effects has that inner/outer key... but I'm not for sure that you can assign tracking data to individual points in AE. And, generally speaking, inner outer key sucks for 99% of all shots...
But all in all - these products offer success in only certain situations (they are all more automated keying rather than true roto).
You'll be hard pressed to find a competent substitute for a manual roto, but the person who invents it is sitting on a goldmine!
Re: Any intelligent masking tools? by Tatha on Feb 24, 2004 at 11:32:33 pm
Thanks for the info J Bills. I shall check the tapes again. Am I right in assuming then that a different tracker can be applied to each of the 4 vertices of the spline thru the roto palette? Applying just 1 tracker to the whole spline will not work as the four points will, in real life cases, move differently relative to the camera (changes in perspective) and will require different trackers. But maybe this is easily possible after all.
I believe Combustion's trackers can be applied to individual mask vertices. But I don't have the software.
Yes, I've tried the Mokey personal learning edition and couldn't really apply it successfully. I think an ideal solution would involve keying as well rather than splines because the edges of the subject could easily be furry, semi-transparent and so on, where splines may be less useful. In fact, in most of the cases I'm talking about, Mask Pro for Photoshop does a beautiful job of local keying at the edge as you paint roughly around it. Keeps wispy hair, fur etc. Little effort in roughly outlining with a large brush. Only problem is since it is not a video format, there's no keyframing possible of the brush strokes. So as the subject moves, changes shape relative to camera etc., you can't apply any one Photoshop action to the entire .TIFF sequence, say. Every frame has to be worked on. That is the problem.
Re: Any intelligent masking tools? by Tatha on Feb 24, 2004 at 11:38:54 pm
Actually, further to my last post, its obvious that Commotion can apply trackers to individual points - corner-pinning is just about that! The question is - does it do this only for pre-defined commands such as corner-pinning or will it accept this procedure when applied to random mask vertices (of the user's choice) from any organic-shaped rotospline. E.g. After Effects will do corner-pinning. But it won't do this latter task.
Re: Any intelligent masking tools? by Matt Silverman on Feb 25, 2004 at 3:02:31 am
The tracker will do exactly what you are asking. Each point on a rotospline can be assigned to a specific tracker (or tracked data can be assigned to the whole shape). The problem with this technique is that the ideal place to add a rotospline point usually isn;t the ideal place to track. This is why the tracer in flame didn't work for J Bills, but works amazingly well when you see discreet demo it on "perfect" footage (ie. demo magic). Usually in flame if I think I can track a simple shape I put the tracking data on an axis (ie. null), then subtly tweak the shape. You can do this in AE as well... instead of masking the layer, add an adjustment layer above your layer and draw the mask in the comp window. Then track the BG and apply the data to the adjustment layer's position. Then subtly keyframe the mask, and use it as an alpha track matte.
I don;t know which tape to check out, but it is the Tracking Rotosplines lesson with a printing press.
As for automated roto tools, nothing really works well. Mokey is great at automatically cloning out areas, but isn;t too hot when you try to use it for masking. Ideally, mokey wouldn;t extract a matte, but would rather create a new editable rotospline. Ultimatte used to make a motion version of Knockout (ie. Corel Knockout for POhotoshop), which worked pretty well from what I heard... however it only ran on IRIX and was about $12k. There is new technology being developed at universities, as you mentioned. However, the ones that seem like they can accurately extract edges do not have user-friendly techniques for generating the garbage matte, resulting in more setup time than it wold have taken to manually roto the shot. One university which has done quite a bit of this is going to be presenting new roto technology at Sggraph this year, which they claim will revolutionize roto... hopefully it will be real.
Another related query by Tatha on Feb 25, 2004 at 11:05:08 am
Matt,
thanks for all the info. I'll certainly try these options. Actually, it would be nice to have a brush which keys keeping hair, transparency etc. like the Knockout version you mention. I remember from your training tapes that when you were extracting a giraffe's neck & head, you first used keying to get the furry edges right and only then used roto to complete the mask. So, however automatic, splines will remain problematic for complex subjects. I was recently extracting a Christmas tree from a busy background in order to insert an object behind it (which could be partially seen thru the furry tree.....the camera was not static....but I could successfully track several points on the tree as the camera moved....if only I could apply the trackers to Knockout/Mask Pro brush strokes.....
Anyway, moving on, 2 slight digressions which could be relevant here:
1) Is it necessary, when using high-end keyers such as Keylight, Primatte or Advantedge, to do the routine preprocessing that is advisable with more basic keyers, e.g. look for the right high-contrast channel, apply levels appropriately etc etc.? Or, am I correct in assuming that the sophisticated keyers just do these internally anyway and there would be no benefit from checking the key applied to various channels or pre-processed in various ways every time something has to be keyed?
2) Anyone use DVMatte from DVgarage? Many have posted positive comments on this implying that this is now there only keyer and so on. It gets quoted as "best of the bunch out there" and so on. I'd have ignored these if the creator wasn't from ILM. What is never clear from the reviews is whether the "bunch out there" comprises only built-in keyers or what. Basically, my question is - as an owner of Primatte (thru Commotion) & Keylight (thru AE 6.0 Pro) is there anything I could gain from DVMatte?
Re: Another related query by Matt Silverman on Feb 26, 2004 at 2:57:56 am
[Tatha]"Is it necessary, when using high-end keyers such as Keylight, Primatte or Advantedge, to do the routine preprocessing that is advisable with more basic keyers, e.g. look for the right high-contrast channel, apply levels appropriately etc etc.?"
These techniques are essentially producing a luminance extraction. Instead of simply using a desaturated clip to derive the matte, you look at various channels to see if any give you a better head start. FYI, another great plug-in to help with this is ReVisionFX Smooth Kit... they have options to seperate true luminance/chroma differential (ie. YUV). Primatte, Keylight, Ultimatte, and other "chroma keyers" are basing their extraction on color instead of luminance, so the channel shifting process is not necassary. These are really designed to handle green/blue screen extractions, and usually luminance keys work better when dealing with extractions from a regular shot.
[Tatha]"2) Anyone use DVMatte from DVgarage?"
I have used it and it works great. I wouldn't think about bailing all of my other keyers for it, since you can never have too many keyers. I have had shots where Primatte, Advantedge, and Keylight all failed, and AE's built-in Color Range keyer saved the day. I personally use Primatte for the majority of my keys, and turn to the others when it fails. DV Matte Pro is very good at dealing with 4:1:1 DV source, since it does a luminance key for the edges (4:1:1 has a full range of luma) and a chroma key for the base, and combines them together into a final key. I recently tried it on tough underexposed greenscreen shot on film, and it choked like all of the other keyers... I extracted it in minutes using the new flame Master Keyer. Master Keyer can probably tempt me to throw away all of my other keyers, but I can't carry around a flame like I can my Powerbook, so I will keep the others. MasterKeyer pulls a base key, then you can add three secondary patches which all get processed individually then output to one final matte. The interface is really exceptional, and I was able to pull the final matte in about five minutes (had about 80% extracted with abother two days of cleanup in Commotion). What was even more remarkable was that it rendered the 12bit 2k matte in 17 seconds on our new 4 proc SGI Tezro. =)
And regarding the "ILM" thing... it was actually developed by a talented programmer named Ben Svyersen (sp.), then Alex Lindsay (former ILM artist) took over sales/marketing/distribution when he saw how cool it was.
Luminance key by Tatha on Feb 26, 2004 at 10:55:50 am
Thanks a lot once again Matt. Your knowledge is truly amazing!
One question about the channel shifting thing though - I now remember from the technical appendices of the respective manuals that Primatte, Keylight and the like use various processes on color information (Primatte's use of 3D color space and concentric circles etc.). But I didn't realize that these were not true all-purpose keyers at all in that even if a luma key on a particular channel, after boosting the channel's contrast , would yield a better starting alpha, these keyers will simply ignore that and keep using a standard color-based process only. Is that a limitation of most of these commercial keyers? Of course, as you mentioned, DVMatte Pro does this 3-pass procedure which uses Luma information, but it still uses just one standard process rather than any conditional branching depending on which methods seem to work better with the given footage.
Matt, just to make my objectives clearer - I am part of an indie team in the UK and we don't have the budget to fork out our small VFX work professionally. We manage with self-learning Premiere, AE, Commotion, Maya and so on, sometimes using Brian Maffit's tapes where available, of course your own and sometimes the Meyer's etc. We have a number of shots to plan, where because of the wide field of view required or because the action will require the subjects to move considerably during the action, a properly lit greenscreen of the required size is looking infeasible. Either the quality of the greenscreen has to be quite poor with just ambient lighting or one would have to be comfortable shooting against any uniformly colored background (a white wall say) and be confident that the subjects can be extracted later. Hence, I'm looking into both keyers and roto tools which might help with about 10 mins or so of this type of footage (not at a stretch but scattered throughout the script).
Thanks for the heads up on the SmoothKit plugin. Shall definitely try it.
Re: Luminance key by Matt Silverman on Feb 27, 2004 at 2:41:17 am
There are three main types of keyers... luminance keyers, color keyers, and difference keyers. From my understanding, the keyers you mentioned are color keyers. Even if they are using 3D space, what they are talking about is RGB defined in three dimensions. This is a pretty complex process and I'm not an enginner, so here is the short answer to what's going on... let me think back to Puffin's whiteboard after we saw Discreet's 3D keyer explaining the "golden nugget" theory...
Normal keyers simply plot the RGB values linearly. If you were to click a pixel to define for your key and then display it in 3D space with red on one axis, green on another and blue on the last, you would have a single pixel sitting inside this three dimensional space. When you add "tolerance" to this, you are spreading the RGB values in every direction resulting in a cube. Adding softness to the key, you are rounding the edges of the cube. Due to abnormalities in most blue/green-screens after they are lit, instead of a rounded cube what you really want is something like a nugget of gold. In order to produce this golden nugget, you need to do the key in 3D space from the get-go. This is what keyers like Primatte and Discreet's Master Keyer, Modular Keyer, and 3D Keyer are essentially doing.
Many times, you can combine multiple keys to create your final matte. If you find that primatte does a great job but leaves edges bad and a few holes, you can then use another color keyer (or primatte again) to fill in the holes, then use a luminance key to get the edges. You can then use lighten or darken tansfer modes to combine these mattes together. Your goal is to create a high-contrast matte, so you gotta do what you gotta to to get to this end.
If you had a big budget, you can get a big soundstage and cover the walls in green... icheck out the Yahoo Critters spot on my site (www.phoenixeditorial.com). We needed to have a soccer match against green, and set up a 60ft high screen by 60ftx60ft. It was huge. If you don't have this luxury, then your best bet will probably be some sort of luminance extraction and/or rotosplines.
Re: Another related query by J Bills on Feb 27, 2004 at 9:25:14 pm
>>MasterKeyer pulls a base key, then you can add three secondary patches which all get processed individually then output to one final matte. The interface is really exceptional, and I was able to pull the final matte in about five minutes (had about 80% extracted with abother two days of cleanup in Commotion). What was even more remarkable was that it rendered the 12bit 2k matte in 17 seconds on our new 4 proc SGI Tezro. =) <<
Matt - I'm a little unsure - is the master keyer a new keyer or a reworking of the 3D modular keyer? Does it still handle patches the same way as it did before, or has that changed?
We're still running 8.3 until we get the green light that 8.5 is stable (heard some reports that it had some initial bugs, so we didn't upgrade right away), but your post has piqued my interest. I do apologize for posting this in a commotion forum.
Re: Another related query by Matt Silverman on Feb 28, 2004 at 3:08:51 am
It is a new keyer that they first showed at NAB's user group last year to a lot of "ooohs and ahhhhs". It is a lot simpler to use than the Modular Keyer. I have a sneaking hunch that it is using Modular Keyer technology in a new simple to use UI.
The master keyer by Tatha on Feb 28, 2004 at 3:50:43 pm
But I guess Combustion still hasn't inherited this keyer. Otherwise, Combustion is known to include the Flame keyer. But that must be the previous one.
Re: The master keyer by Matt Silverman on Feb 28, 2004 at 6:40:07 pm
It's a bit deceptive... Combustion uses the old, old, old, flame keyer. It is a pretty good linear color keyer, but nowhere as good as Ultimatte of Primatte. On my old flint I would always use a Primatte spark for keying. Flame added the 3D keyer back in '98, and then followed that up with the modular keyer. Then they added the Master Keyer in the most recent version. I highly doubt that any of these new keyers will trickle down to Combustion in the near future. Discreet's advanced systems make them a LOT more money than Combustion, and they don't want to piss off these users.
They also say that Combustion uses the same color corrector as in flame, however Combustion does not have the Color Warper, which is the super-powerful primary/secondary CC in flame.
Not knocking Combustion here, just trying to point out the facts. C3 seems like a great tool... I was blown away by the speed of the tracker on the g5... seemed like flame speeds. And the tracker is indeed the same as in flame, however, post tracking control in flame is superior. Maybe expressions in C3 could help solve this, as I have found expressions can in After Effects.
Speaking of which, I will be doing a one hour theatre demo at the Adobe booth during NAB on Monday and Tuesday, 12pm to 1pm. I'll be showing some advanced tracking techniques for keyframing offset values of tracked data with expressions, as well as showing some tricks for tracking masks in AE. And some recent work... I showed some of tis at the BAMG meeting at MacWorld, and everyone was really psyched on these tricks. Stop on by...
Re: The master keyer by Tatha on Feb 29, 2004 at 9:27:43 am
Don't know whether Matt will read this anymore. But this is in response to his NAB show event. Just wondering whether any tapes / books / tutorial CDs etc. of these new techniques would be available later for a wider audience (e.g. like me who doesn't live in the States). I'd certainly be interested.
As with everyone else here, I would like to express my sincerest gratitude to Matt from whom I've learnt much within a week of interaction here.
