Having more testosterone is less indicative of a positive boost in sexual drive and more indicative of an unhealthy boost in sexual drive, such as rape or child molestation. Fortunately, baldness is not directly linked to testosterone, seeing as old men with little testosterone tend to be bald. Also, 5-alpha reductase is the real culprit in balding. Sex isn't very important to me, and I certainly don't want people thinking my bald head means I'm a hump machine. I do not want to attract people who are attracted to me for that reason. Old wive's tales only make society dumber.
A man's rate at converting testosterone to estrogen creates his hormonal balance, which leads to sex drive.