Who says I'm not there to rub elbows with the nobility and dance? That's exactly the reason I want to go to an Orlesian ball! I want a night off from being The Inquisitor and celebrate!
Though I would love the chance to let my Inquisitor relax and have fun--with companions (like in the Citadel DLC)--I don't think I want to do that around Orlesian nobility. Too snotty, and well, vicious. Rather throw a party in the tavern at Skyhold. 
But the option to completely scandalize the nobles however... that's the tempting part. 
They should wear their armor then, not some prissy made up uniform with no practical purpose or protection.
Also that outfit is hardly unisex. It's based on oldschool men's uniforms and even accentuates and widens the shoulders.
The ball outfit is about as unisex as this:
No, I agree armor definitely makes more sense. But that would also project that the Inquisition isn't even trying to compromise on or cares for Orlesian customs which is not how you garner politic favor. I see it as a matter of give and take. "I came here on your terms, but I will not bow to you," if you will. But again, that's my take and why I think it's appropriate.
As for unisex clothing, and--mind you--I'm not saying it's right, but the world is very cis heteronormative and because of that every bit of unisex clothing I have ever seen has used the male template. So, yes, it still falls under the category of unisex. If I said androgynous, then I would expect a mix of traditionally "masculine" and "feminine" qualities, but the uniform is not as such.