Really this shouldn't happen but I managed to make it happen. Until this
comes up again in a way that matters, let's just fix the LINQ crash from
`SingleOrDefault`.
I've tested this to work as expected in the broken scenario.
The whole component is pointless so I'm just going to nuke for now I
guess. Kind of makes the whole refactor effort pointless but oh well?
To expand on this, the implementation was actually incorrect as pointed
out at https://github.com/ppy/osu/pull/18819#pullrequestreview-1017886035.