Unless I'm misunderstanding you, this is normal. Without a camera in your scene, AE acts as if you have one essentially dead center or looking flatly forward. So when you move your layer away from the center, it's perspective changes as it would if you were to add a camera to your comp and then leave it's default settings as is and not move it. Again, not sure if I'm misunderstanding you. If so, I apologize.