I find what works best for me is where ever i find the rough stubble where i rub, i slide the razor in opposite direction. so effectively going against the grain. especially on the back on the head where it grows in all different directions!
I also try and go over the area in least amount of times, like your self. Hope this helps.