I found a solution.
It was definitely the time remapping. When I turned it off everything worked normally.
So I instead rebuilt the face rotation using a joystick.
It wound up working better anyway. When I had it time remapped it was just the head looking left and right.
With the joystick I was able to add an up and a down so now I have a full range of movement with the head.
I can now add a bunch of nuance with how the head turns. Much better.