If the Dalish historical account about being invaded for their refusal to convert is right, I don't see how that makes them the aggressors.
Those same accounts that ignore elven slavery? And also the fact that part of their parthenon consists of eldritch horrors? And don't acknowledge the cities they took over or people they killed in their war against the chantry?
By the sound of it, their homeland of the dales was a pretty defensible place, they could have bunkered down and weathered orlais's advances for some time, even having a steady supply of food and water from the forests, orlais would probably not have been able to call an exalted march if they'd stuck to that plan. Given that, the only reason to attack red crossing and all the other cities, was to expand their territory, not for any sort of defensive purpose.