I don't think this is necessarily a final solution (as this means all
HUD elements are adding overhead even when not visible), but this will
make the implementations much easier for the time being.
I've checked and can't notice any perceivable overhead in profiling so
we should be fine for now.