Not really sure of the best way to handle this in general. It could be
argued that this should be a `Component` type and the bindable bound in
`LoadComplete()`...
Closes#20994.
I haven't considered how this affects performance of difficulty
calculation or otherwise. Seems like a sane initial fix which we can
iterate on going forward.
I've tested using the scenario in the linked issue.
I'm not going to add test coverage because
[BeatmapConversionTest](https://github.com/ppy/osu/blob/master/osu.Game/Tests/Beatmaps/BeatmapConversionTest.cs)
should correctly cover this scenario once we are serialising control
points as well.