I run a small gaming site and have been debating the same thing for a while. Have come to the conclusion that unless there is a strong need to upgrade, I don't, because of the same issues you listed (mods, CSS tweaks, etc.). I don't want it to turn into more work!