Like Beerfish said, Co-op gameplay has a huge difference in general attitude from regular pvp gameplay. You aren't fighting each other, you're fighting together. From my experience in ME3 mp, people bad mouthing each other has been really rare and the 1% consistent trolls must be some real douche bags! Pvp though is a whole another level, and bad mouthing and trolling are part of the game. Think of it as a sort of mental warfare in a digital era.
That being said I wouldn't wanna be moderated in either version of the game in any way, hell no! Besides what I feel is an unrealistic and unnecessary waste of manpower, I don't want another hall monitor in my game.
All in all, while some general tips about the benefits of not being a jerk are nice, there isn't much else that needs to be done. It's just the nature of the games. Think of it this way, it's kinda like boxing and golf. People getting mad in golf is rare but it is to be expected in the latter because that's just what the game is about.
Edit: Oh I forgot to mention the limits being that as long as it stays within the game and doesn't lead to any actual harm it's fine. You might've heard about people getting DDOS-ed or swatted and getting threats in real life. Now that's just wrong and fucked up!