What you need to do is use a couple of layers.
On one layer, you have your background KEYED out using Keylight or whatever.
On a layer just above that, you do your ROTO work on the shirt.
That should work well for you.
Reel Inspirations -
http://www.reelinspirations.com
Commercials, Dramas, Image Pieces, Documentaries, Motion Graphics