I would argue that they are. The misunderstanding is the widely held belief that there is some "right" way to live.
We're all the victims of that misunderstanding.
That same logic could be applied to a multitude of issues and actions, but human culture is composed of varying things that we considered to be right or wrong. I'm not saying polyamory is bad or wrong, but you could literally use that argument to defend a serial killer.
The fact of the matter is, if the polyamorous individual was aware of their lifestyle before entering into a relationship and their partner is hurt due to the misunderstanding, the monogamous person is more of a victim. To argue otherwise would be akin to saying that a murderer is a victim because he has to suffer the consequences of his actions.
But people get hurt not because the polyamorous behaviour is somehow wrong, but because it's unusual. People who aren't capable of polyamory aren't going to he compatible with the lifestyle, and they don't know how to deal with it.
Education would help. Or just not making assumptions about people. A person is complex, and it benefits all of us to think about each other complexly.
I just want to say, again, that I don't think polyamory is wrong. It's not for me, but I know it works for others. As far as I'm concerned, if everyone involved is okay with it, it's none of my business. You do you, polyamorous peeps.
Unfortunately prejudice and assumptions are a part of natural survival instincts. People are going to expect what is considered the norm. It's not your fault if you're surprised to hear a giant man speak with an unnaturally high-pitched voice. But education does help when it comes to understanding a group of people and not hating them just because they hurt you or someone you know.