Use a video layer like a shape mask for an effect
Ok, I'm a novice at AE, but I'm decent with photoshop. I have an effect I use often to add clarity and even depth when the image is a bit flat. I essentially use the "create a sketch" effect to detect the edge detail, then adjust the detail to what I want to effect, and then turn it black and white. I then select all the grey and black areas with the color selection algorithm, and create a selection mask channel from it. I add adjustment layers that darken some areas at the edge and transition areas to "sharpen" the focus there, I then invert the mask and apply a light blur everywhere else, but I use the burn dodge tools to adjust the mask so the blur fills more as the depth goes further out, it creates an almost creepy bit of depth, and done right, it is a big seller. It can even work some magic on less than perfect focus.
Here's my problem, I'm great at photoshop, terrible at AE. I would like to use a video or precomp layer like a mask. I can get the edge detail and sort it out so that I have the detail in a masklike black and white, but I cannot use it like a mask. I can apply the appropriate blend modes, but I cannot selectively apply it. Is there an opperation that would allow me to do so? If not, I suppose the only option would be to create a black shape alpha, fill it with white, and have my video layer apply that way, using a blend mode, correct?
I may try this with an output of the sketchmask to a folder of images so I can make adjustments, then I'll apply all the images as a video layer and go from there. I'd rather be able to mask this in straight out, but I can work with a shape if I must.
You should be able to get there in AE. Pshop and AE are very similar - Pshop got its layering system from AE but alas AE has yet to get its paint from Pshop (how long have we been waiting?).
Anyway - the only real difference is in the track mattes in AE and the precomps instead of folders.
To use the track mattes, a masking layer sits on top of the layer you want to mask. The track matte layer becomes the mask (and is usually turned off) and you can choose either to use the trackmatte layer's alpha or its luma levels if it doesn't have an alpha. You can reverse the masking as well.
Depending on what version of AE you have you may not have any folders to work with. (so that you can pass blend functions through from whats inside the folder). To do this in AE you may have to precomp the layers in question. That precomp becomes a single layer in the outer most comp, and that precomp gets rendered before it gets layed on top of the rest of the layers in the outer most comp (AE is more like Illustrator in this way). The problem is any blend function you use on the precomp in the outer comp is on the full render of the precomp. And any blend function you use inside the precomp is only good inside that precomp and doesn't always work if that precomp is on nothing because you are trying to keep your matte ("Screen" for example doesn't work properly if there is no background to screen onto).
What might be best is, to have a go at it in AE and then post the project and we can have a look and suggest a way to organize it so it does what you want.
With the exception of some fx that might not be in AE, that you do have in pshop, all should be doable. Even things like running the dodge tool along a path can be done in AE but in a different way (the path becomes a mask that you apply a levels filter through)