I have two layers of the same clip that I am trying to mask different parts of. When I apply one mask it's fine, and as soon as I apply the second mask it changes the mask underneath it to match. If I hide one track or the other, the masks act correctly, but once they are both on they are mimicking each other. I have tried nesting and it seems to work every now and then, but if I move the clips a frame or two it snaps back to this glitch.
There are blend modes for opacity but that's not doing it. My work around has been to Render & Replace the clip and then stack the layers. I think it is confusing itself when there's an exact duplicate of the clip on top of the other. The R&R seems to break that connection. A little annoying but it's working!