I'm involved in a project which is being raytraced with refractions. Maya Software renderer.
For instance, imagine a jar with whiskey in it. You can still see through it, but areas of the matt will more or less trnsparent. I would like a mask that allows me to comp this. What I am getting so far is solid matts, or not-too-useful matt gain-ed matts. I'd love to have more gadated masks through 'fluids'. Im not using fluids per se, the renders will be for magazines.
Is there a way to get the matts to obey some transparency linked to the refractions in one pass?
Actually, I found a 2 pass solution. The reason my matts were coming up solid, was that I had a textured shere to act as the reflected/refracted background. Even with primary visibility off, the object made the matts opaque. Solution is to derive a mask in another pass or render layer without the environment object.
It's an expensive solution since I'd have to RT 2 twice.
Yes, since your refraction is refracting values other than camera background color they will have an alpha relative to the transparency of the pixels being refracted. And that's only because the transparency allows background color to show through. You're two pass solution is probably the only solution in maya. In mental ray and other renderers you might be able to apply a rayHit shader to the sphere that would return the color in refractions but an alpha of zero. There's just no control for that in maya.