Considering that the Qunari/Kossith/Iron Bulls ppls-WHATEVER are possible racial science experiments gone wrong (right? *woof*), I can see this as a thing.
I'm going with...the Kossith were the ancient initial forms of the Qunari. Remember those dragon-guy paintings? They might be hornless, but they look far more like dragons than current Qunari. They were perhaps favored of Elgar'nan for their inner rage and power. But at some point the Kossith had a schism and split, following arguments over what path their race should take after the fall of Arlathan.
One continues with uncontrolled power, rage, barbarism. One says mastery over themselves, bridling the fury, and serving for the greater good of their race is a better path. Somewhere along the line, the Qunari split becomes less dragon-like over time and eventually make their way to Par Vollen (any any other places they've conquered) in order to put additional literal distance between them and the Kossith. Could be they were kicked out of their homeland, or seriously threatened by a potential Kossith invasion.
Iron Bull mentions briefly that he doesn't think Qunari and Kossith look much alike. That tells us that either he personally hasn't seen a Kossith, or the Qunari as a whole haven't seen Kossith in a few generations. If they did, you'd think there would be better records than "we had to leave our homeland because reasons" like Iron Bull tells you.