A room scene that I'm currently working on is rendering out fine except for some sever jagged lines around the windows which the vray lights are coming through. Everything else in the scene is nice and smooth with the antialiasing filters I have, but no matter what filters/settings I keep tweaking the black frame of the window that sits against the vray light continues to come out jagged.
In the area where you see Jagged lines, in the vray frame buffer can you right click and roughly find the brightest pixels? If it never goes above 255 it might mean you have clamping switched on in the vray settings. With clamping off, if the brightest value exceeds 255 by a large amount it might mean that although you may have low level lights, the camera exposure might be wrong, which could still make some lights appear over bright.
Just a thought, are your window frames reflective at all? If so they might be reflecting the Vray lights causing you to essentially see the reflection of something that might be too bright for your exposure settings to deal with. Try setting the vray light to not be visible to reflections if this is the case.
Let me know how it goes.
3D animator and VFX specialist