I don't see it as a practical problem, because no one will touch you.
Nazis will think you adimire them, and the others will not buy the thing sly guy/nazis or, if they will, they will be also be intimidated and will stay away from you.
Maybe tre problem could be psychological for you, because you don't want to be considered a nazi, but if someone judge the book by the cover... you know... better loosing than keeping such a person.