I use baby oil and don't have a problem. I have been "blessed" w/ problem skin all my adult life so I don't think that the oils in and of themselves cause blemishes. What does cause problems is dirt, and I wash my face and dome at least three times a day, before heading to the gym in the morning, showing after and before bed. Since I don't have dry skin problems it works for me. Keeping the dome clean keeps it clear.
Baby oil is a big no-no for me, unless I purposely want acne all over my scalp.
Stop the baby oil and try shave secret.
I always use Shave Secret....I tried the baby oil about a month ago and was not impressed...