We're all ideological minorities, but those of us who are white can blend in easily, until we start to speak our minds.
Ian and Mark have said that New Hampshire people don't have a problem with black folks in general, but keep in mind that this is a state that is well under 1% black. I would assume that one would get used to looking a bit different. I've never been to NH, so I can't speak from any experience.
If more black folks realized how much the state has victimized them, more would be libertarians. Unfortunately too many see the state as benefiting them, and many white folks see the state as benefiting black people. It's what the state does best, it harms people while making them think it is helping them.