Unlike other QuickTime apps, Motion "filters out" VideoOutput sizes that don't match the current project size - which means if you're trying to work on a non-standard frame size (e.g. 320x240) Motion will eliminate ALL of the possible settings!
If you're using one of the usual "broadcast" sizes (720x486, 1920x1080, etc.) it should be there.
Actually, I was wrong: Motion removed the "filtering" behavior I mentioned, so on my system I see all of the available Kona Video Output modes.
Just to make sure we're talking about the same thing: I'm looking at the Motion Preferences dialog, under the "Output" tab, in the Video Output popup. I see 10 different "AJA Kona" output modes, including four flavors of 1080).
If you're not seeing the same thing you might want to give AJA Customer Service a call.