If a couple has an open marriage, I wouldn't even consider outside partners to *be* infidelity. By the terms which two consenting spouses have chosen to define their marriage in that scenario, no one's being unfaithful. ;) Conversely, even though it's a dire truth that spousal abuse is socially acceptable in many cultures (even some corners of own), to my mind the very idea of it is akin to infidelity in the broader sense of the word (i.e. disloyalty, breach of trust); trusting or having faith in a partner who hits you is a complete anathema to me. So, casting the first stone n' all that...
What I'm trying to say is I'm with you on how one loves or is loved is, yes, totally their business and violence is NEVER okay. That said, even if such circumstances are not in play, plenty of people make lousy decisions that hurt rather than help their circumstances in a marriage; playing in that sandbox for dramatic effect in a show is not something I am compelled to write off unless it drags out too long or the characters aren't allowed to grow as a result.
(I brought it back round to topic, I did! ;) )