This may be a stupid question but I thought I'd just check incase there's a solution that I don't know about.
I wondered if there's a way to group anchor points so that they can be easily selected.
This example images isn't that complex, it's just a reference, but if there were a set of anchor points in the middle of a group of other anchor point the only way that I know of is to select all of the anchor points near to the ones you want and they deselect the ones that you don't want.
Any you can't click and drag where there is a shape because dragging just moves the shape.
If you could select the group of anchor points that you wanted in one click it would make things so much easier when working with more complex designs, don't you think?