I flipped the normals and enabled Backface Culling. Everything works as it should be. Although I ran accross something. At some angle a back face would be selectable if facing towards the camera. It's not a big issue since moving the camera just a hair would get around that.
But: Is there a way to turn off the selection of backfaces overall?