after effects

what version of Ae are you using? the refine edge tool was introduced in Ae cc (2013) and you can select it by clicking and holding on the Roto Brush tool or alt clicking on it


here is an excellent free tutorial:
https://www.lynda.com/After-Effects-tutorials/Refine-Edge-tool/124400/12992...