There was a time in my life when I didn't care if they were married or not. But since coming into "old man" status, my view is more on the side of thinking about the guy they are married to, and their kids (if any). I feel trying to bed a married woman is taking advantage of them or their circumstances. When you are married there are always problems .... and you work together (or not) to solve them and make the marriage work. You both don't need any outsiders adding to it, or breaking up your home.
I am not criticizing anyone who feels the opposite, at all.