"Lore justification" is a lazy excuse for being exclusionary. Where's the lore justification for the people of colour to be a separate group in the first place? There isn't one, but I don't see you complaining about that.
I'll pull this part off to address.
Basic genetics tell us this. These nations have existed for thousands of years. By looking at it through the myopic and bizarre filter of our world today, you miss basic common sense. Physical traits are genetics. The only way people can look different enough to qualify as a separate race is after centuries, if not millennia, of independent breeding. In today's world of fast travel and increasing diverse cities, people of all ethnicities life together. And, if we continue doing so for another 1,000 years, then people will all begin looking the same through dozens upon dozens of generations of people having babies together.
The same would happen in Thedas. The "people of color" in Thedas live on the island nation of Rivain. Why? Because if they lived everywhere, equidistant from the "honky" looking people, as you describe them, then between even the time of the last Blight and this one, there would be no more dark skinned people, nor lilly white honkies. It would become a mix of Mediteranean looking folk, similar to Duncan.
That's not to say Rivain people can't travel - Isabella and Vivienne clearly have. But for there to be people different appearances that are analogous to our real world ethnicities, then geographic differences (or vey strong socio-political ones, like what we see with the Alienage elves and how cross race relationships are frowned upon) are required.
Now... could Thedas have been set up as a culturally varied, rainbow ethnicity world where all real world people were reflected equally? Sure. The questions of basic genetics would be raised momentarily, but if the entire world was designed that way, where any couple who had a baby had an equal chance of popping out a ebony baby as a lily white one, then it could be ignored.
But Thedas was not designed like that. That ship has sailed. If you say keeping the world as it was designed is racist, then you are saying Bioware was racist with their world design. And, you know what? Maybe they were.
But that ship has sailed. And introducing further retcons and lore damage like the Qunari is not the answer to anything.