For me it's against the grain all the way. Going with the grain is just like not shaving. Doesn't get it close enough. There are those spots, like on the crown of my head, where you gotta go in all different directions just to try to get it smooth.
With on the first pass, against on the second.