Pushing this out with zero testing as a "probably fixes"
https://github.com/ppy/osu/issues/33108.
Previous logic would mean that the start value of the fade-in may not be
zero depending on current state, leaving the final alpha state incorrect
after a rewind beyond the start time.