I think you can’t reference with the same expression to the valueAtTime from another second. Because AE reads the expression every frame new.
I would do it like this:
//var scale4seconds = calculations which should affect only the scale before second six
// scale4seconds + newCalculations
// do the calculations which should only affect the scale before second 6