try using a roughen edges effect with these settings:
Edge Type: Roughen
Border: 50
Edge Sharpness: 0
Fractal Influence: 0
or (multiple effects needed)
- Add any Blur ('Box Blur', 'Fast Blur', 'Gaussian blur', whatever)
- Add a 'Levels' effect where you set "Alpha Input Black" = 128
- Add a "CC Composite" effect
The blurriness of your first filter takes care of the feather radius
This has some side-effects if the layer you are starting from allready has a soft-alpha (but I'm guessing it doesn't in your case)
BTW: To easily mask the entire layer, select the layer, then double-click on the rectangular mask tool in the toolbox.