You can create square/rectangular Shape layers with round corners using the Rounded Rectangle Tool (use the dropdown next to the pen tool) - set the roundness in its Contents > Rectangle 1 >Rectangle Path 1 > Roundness. You can use these as Track Mattes if you need to round off the corners of other layers, images etc.
Use the Channel Blur effect. Blur just the alpha by something like 100.
Then, apply the Levels effect. Change the Channel dropdown at the top of the Levels effect to Alpha. Crush the levels effect so that the three triangles are somewhere in the middle. Alpha Input Black to 132 and Alpha Input White to 142 - for example.
Those values will give you a very round corner on your image. The amount of blur in the Channel Blur effect will determine how rounded your edges are.
- The Great Szalam
(The 'Great' stands for 'Not So Great, in fact, Extremely Humble')
No trees were harmed in the creation of this message, but several thousand electrons were mildly inconvenienced.