Yes, you can get irritation from the brush, but it's not the usual culprit. Pure badgers aren't known for being the softest, but unless you have a particularly bad one, I don't think that's the problem.
Usually the problem is either pressure, too many passes, or shaving unlathered skin. If you suspect the brush, try using it to build your lather in a bowl, but applying the lather to the skin with your hands. If you still get irritation that way, then it's not the brush...