Please file a bug to Autodesk
Bug being as you describe that selecting a mask point in RE:Flex and pressing the up and down arrow does not move the point. This works in other Combustion tools like Draw Mask and would be expected to work in RE:Flex as well (same polygon toolkit).
* Moving the tracker position within RE:FLex works but that's of no help.
* If you select the point in the timeline, there is a way (pressing shift) to increase the increments but apparently no way to decrease it so it so it moves by sub pixel increments.
The only I can think to work around this today is by working on a zoomed up image (pressing the "+" zoom-in button a couple of time)