First, as an amateur psychologist myself, an important thing to remember is that because I logically deduce something doesn't mean it's reality. Further, I may be wrong. It's very important to remember that, I find.
As for your last phrase, I see uncomfortable analogues.
Lets reframe it:
"Why are you against improving racial representation? I hate that we don't!"
"Why are you so set on changing racial homogeneity? I love it!"
We aren't many steps away from:
"Why are you against equal rights for all? I hate the inequality!"
"Why are you against me having additional rights and authority? I love it!"
You misunderstand. Those paraphrases at the bottom of my argument were abbreviated versions of how each side perceives each other, in a distorted, instinctual manner.
The SJW side can find it baffling that people would oppose moves to be more inclusive, since it is a positive goal. With the dudebros see it as change that could risk reducing their enjoyment of video games for a goal they don't see much value in.
Your above response makes it seen like those against inclusion just for inclusion's sake in an entertainment product are against it because they prefer a white and male dominated game. I don't think that is a healthy mindset for anyone to have, as it presumes that the person on the other side of the argument is racist and/or sexist. Similarly, I don't think the dudebros of the world have a problem with fighting for equality or social justice, but just that it should be a secondary goal to the development and enjoyment of the game itself.
I'll use an example. The inevitable "Asians in DA" thread that seems to reincarnate itself roughly every three months. In that thread, people request that there be people who, in appearance, represent people of eastern Asian descent in the DA setting. In this thread, I always respond back that there are no known peoples who appear "Asian" and that if such a people were to be introduced, they would need to have some form of explanation (migration, travelers, refugees, invaders, what have you) to show up in the game world in order to maintain the game's lore, a crucial piece of game development for me personally.
And yet, I am accused, either directly or through implication, of being racist for such a stance - even when I provide quotes from David Gaider that this is the exact method they would use in implementing another group of people who had a similar appearance as other real life ethnicities in Thedas. The responses people give are "why should they have to explain someone who isn't white, that is the very definition of racist," or "so you want to put hurdles up just to keep your white gaming environment all to yourself" or other statements of similar nonsense. I'm not against Asian characters. And neither is Bioware - you all made Jade Empire for Pete's sake.
But as someone trying to preserve the lore and saying "this could happen, but something like X needs to occur, because having people just shoot up out of the ground does damage to the setting and suspension of disbelief for the narrative," something that is part of what makes the game good, I am being told I should put aside my concerns about the game's quality and support the implementation of the inclusive change, because to do otherwise is racist.
That is what people who are against inclusion fear (and sometimes see). That these changes aren't done in a logical manner that provides both inclusivity for more people and also keeps the game's integrity intact. When the cry for inclusivity is louder than the call for good games, that's what gets people concerned.