Raw Footage, No Contract, No Deal

I recently was paid to produce a music video for someone. I wrote up a contract but before filming started the artist refused to sign the contract since we were way off schedule. I'm young and this was my first big production, so I let it go and continued filming (bad move, I know now) since we had traveled to another state. A few things went VERY wrong but we finished filming back home a few days later. I edited everything, gave the artist the final product and he was very pleased. The same day this person finds out that I favor someone's music, but this someone is an atheist. The artist calls me telling bible stories and telling me where he stands on religion. I didn't have to listen to that so I hung up. A friend of mine picks up the phone when he calls back and he also tries to push his religious views on her. She hangs up. When he calls back he leaves a voicemail saying that I was rude and a bad businessperson, and he wants all of HIS raw footage BACK. I purchased the mini-DVs used for filming and I given him the final product on 5 cd-rws in 2 different formats. His lawyer (who didn't give me a name or law firm) called me and I explained to him how this man was proclaiming me a "devil supporter" and said he couldn't work with me so he wanted (once again) HIS raw footage BACK. The artist continued to call my phone and leave voicemails saying he was going to take me to court and he's gone through this before and that I was "messing with the big boys now." After I woke up the next day to a text message from his girlfriend saying i need to give up the footage "before I lose money" and voicemail from 1am from him saying this is his final warning I get my attorney involved. I believe I shouldn't give him anything because there was no contracts and I never agreed to give him the raw footage. He's never hesitated to call me for anything pertaining the project but as soon as he saw something he didn't agree with he ran with it and brought up taking me to court for something that wasn't his. I also found it fishy when he said he's sued a director before over a video, but that director tried to withhold the final product, which I did not. Any advice on what I should do? I do not want anything I've filmed to be re-edited and sold without my permission and I'm just trying to protect my growing brand. I just want to be a filmmaker.