I considered having this just be a straight guard followed by an
early-return, but if that guard ever actually gets hit, things are going
to be severely broken all over, and un-breaking them at that point will
be very annoying, so just going to cross fingers and hope this can be an
assertion forevermore instead.
Previously the logic would fall over under various scenarios due to
applying aggregate logic where it shouldn't be, or vice-versa.
Now the sorting filter explicitly checks the grouping mode and reacts
accordingly. I was hoping we could avoid the sorting filter having any
knowledge of grouping, but I don't see a way around this.