Disable difficulty calculation for set-level difficulty icons
Due to pooling usage, there is no time we need to unload. Switching to DelayedLoadWrapper cleans up the code and reduces overhead substantially.
Closes #10470.