Check out www.askandyaboutclothes.comIt sounds to me like you need to just wear it and get used to it -- then you won't even realize what you're wearing.
My comment was mostly tongue-in-cheek, but I think that it's mostly because straight guys generally don't seek out a career as a men's clothing clerk (sort of like hair dressing). There's probably also a hefty psychological component, as you surmised, where boys are brought up to think that "real men" don't care about looks (after all, they'll get wives who will dress them, right? )