It seems that my nagging worry that mastering is super complicated is true
There are indeed a couple technical complexities you need to be aware of (standards, inter sample peaks distortion etc), but the real difficulty is more about knowing your tools inside out and, even more, training your ears - and this requires time and experience, so
I can’t really rush experience now can I?
No, but you can learn faster if you know what you have to learn and how to learn it - and find the right resources to guide you through the process, which alas requires to already have some basic understanding of the topic so you can weed out the snake-oil guys (as a rule of thumb, you can consider that at least 80% of what you’ll find on the internet is crap and bullshit).
Now the good news is that nowadays you can at least experiment things without having to shell out big bucks, there are plenty free (freeware or OSS) tools that are of professional quality and can run on any average computer. Also, the most essential skills required for mastering - correct use of EQ and compression, understanding of basic psychoacoustic, and well trained ears - are also beneficial to mixing and even composition, sound design and arrangement, so it’s definitly not a waste of time.
As a side question, if I wanted something professionally mastered, I’d have to pay out of pocket for that. In turn, should I wait until I have a fan base large enough that hopefully I’d earn back the money spent on mastering? Or, should I do it anyway in the pursuit of better-sounding music?
That’s a quite subjective question (depends on your goals, wealth, and whatnots) but you can safely consider that you’re not going to have much return on investment (and yes, that’s an understatement) unless you are a marketing genius (and even then…). Also, there’s nothing magical in mastering so you may first want to learn mixing, as it will do much more for “better sounding music” than whatever even the best mastering engineer can achieve.
My 2 cents…