Likewise, I shave daily to avoid the sandpaper texture. But it's a matter of perspective -- it's part of my ritual that I look forward to rather than dread.
Color will develop (even with a little sun) ... just takes some time. And, truly, no one else really cares.