I feel you on fighting the urge to not shave. I like the feel of even 1-3 week long hair, but shaving it can make weather conditions feel really nice. Breezes, rain, wind as you're driving down the highway during the summer...
Funny that you're actually wearing hats more often. I'd never really worn a hat in my life until recently, only recently had I started to like the look of a hat on my head, which was a personal development unrelated to any hair loss revelations. I just happened to start to get into 'em, and now I love wearing 'em. It's just a coincidence that I have no hair on my head now. The hats surely didn't cause any loss and I'm not trying to hide it or anything, even though I'm sure there are guys who try and do just that.
If your car show buds are still hanging on to scraps of hair at their age, I would laugh at any crap that they give you for shaving. Especially if you still have a full head at your age. If they give you crap, you could always ask 'em why they haven't just taken the plunge themselves, if you want to playfully throw a little back at 'em.