Empty Trash, restart, run Disk Permissions/Repair on the HD for starters.
I would guess that either the drive has a problem or there's an application that's hanging it up. Try making sure all the applications are closed, see if it ejects faster. Try on another Mac (if you have one) if it's still slow to eject, then you'll know it's the drive. You could try something like Disk Warrior, but if it's a hardware issue it won't help.