Does the Handbrake application name appear in the menu bar next to the Apple menu, and in the Dock (to at least indicate it has actually launched)? If Handbrake is running, have you tried going to the menu> Window pulldown and gone through the window selections offered- Handbrake, Queue, Activity monitor?
Handbrake is for conversions of DVD and other video source files to MPEG-4 format for iPod, iTunes and mobile devices; really not suited for conversion to QT's for editing in FCP, if that's what you're intending to use if for.
Appropriate and best free appliction for that purpose is the frequently-cited
MPEG Streamclip
