You can use 2 things. First, you can use physics time factor, as you mentioned. Make sure it gets keyframed to 0, not negative. This will stop all movement - there should be no drift. The other thing you can do is turn up air resistance - this will stop particles assuming the number is set high enough.