So, I have been looking everywhere for this transition but wasnt able to find it anywhere, I made a gif of it, so if you know it, please comment. Thanks (Link to gif: https://giphy.com/gifs/Z0lt3SgD2qae4 )
It's hard to see exactly your transition since it is so rapid, but it appears as if the second clip slides in from the bottom with a black border. If that is correct, then the transition you want in Vegas is Slide, with the preset Slide in, Bottom-up. Once the window opens, there is an option to add a Border.
It looks like an overlay mask, because it has a slight radius in the inside corners of the black border. This can be done with Photoshop by creating a PNG file type of a black border and transparent area where the picture will be. This would be set on the track above the picture and made a Parent Motion with the picture being a Child of this so they move together. It looks like as the old picture is moved up, it tilts slightly left and right, giving the impression that someone is "flipping" thru the pictures. This would be done via keyframing the motion as the picture slides up.
A similar effect was used in a sports video I put together. In this, the picture is made to look like a picture is being taken and spit out. The video is at
, starting at the time 0:31. It repeats with each new player intro (every 33 seconds).