It looks like the culprit is the blue channel. This is the problem with compressed footage being used as a source for the key, the compression adds noise to the noise already there from the camera.
But the green channel is quite clean (this is why we use green screen whenever possible).
So here's a file without the usual standard keyer that produces a better result and keeps the shadows.
Have a look and see if that does it for you. You will need to do a little work on the highlights and reflections, but these should be there so that you can pump color from the new background through them as though they are reflecting the environment.
That happens often whenever the green/blue screen video isn't clean enough for your software to intelligently autokey them. Under those circumstances, I will recommend masking them out manually or automatically if your software supports it. Good thing is this specific scene you've shown isn't complex, shouldn't be difficult to manually mask out the entire shot.