I am trying to layer an explosion effect under footage so that it looks more natural and realistic. I spend 3 hours yesterday trying to accomplish this with masks but tnothing worked. Is there any other way to do this?
You might be able to avoid masking and pull a key if the foreground and background are significantly different in color or brightness. If that doesn't work, and if the objects are moving and you need to build a frame-by-frame mask, this is called rotoscoping.
Also, any time you have something bright behind a darker foreground object, it will look more realistic and convincing if some of that bright light bleeds around the edges of the foreground object. This phenomenon is called light wrap. Here's an example of how to create a simple light wrap in After Effects: