I keep trying to use the simple grab tool "Space Bar + click and drag" to move around my image but every so often or sometimes more than that the image zooms out to fit on screen and if I hold the mouse button down I am given a zoom window and can choose where to zoom in to.
It annoys the heck out of me and at first it felt like it would just pick a random part of the image to zoom into.
How am I doing this so that I can stop having this happen?
It's called bird's eye view, actually it's quite useful, but takes time to get used to it. Its default shortcut is H, if you're holding it and release it; you need to change the shortcut to a different key.
It happens only if you keep pressing the 'H' key and the mouse button.
To use this tool more effectively, Press the 'H' key on your keyboard and click on the left mouse button, and without removing your finder on the mouse and from the keyboard, move the bounding box to the position where you want the zoom to happen and release the mouse.