This reverts commit 8115a4bb8f.
Commit was cherrypicked out to a separate pull on a different merge
base, then reverted in that pull, so it should be reverted here too.
Fixes the mod-related overhead portion of https://github.com/ppy/osu/issues/14638. There's still a significant performance issue that will be addressed separately.