It's a little frustrating that the Wardens are so terribly disapproved of in general. Like everyone forgot that they're needed for the Blight should it ever come back. It's why I can never choose to exile them. It just doesn't make sense.
I think it's more "Oh dear god the warden mages leashed themselves to demons with blood magic from killing the non-mage wardens... and you came from the future and saw that a giant demon army killed everyone in Orelais (presumably led by the Grey Wardens.. witnessed if you side with the mages). Oh yeah and in your fade vision you saw the Grey Warden's kill Justy V."
But let's give the Wardens hot cocoa and hugs and hope Cory doesn't wiggle his fingers and take them over.
We'll call you if there's a blight, but toodles do not pass go and do not collect $200.
My Warden was a bro and he's out looking for a cure to the calling. Allistair is on the throne and not doing anything full blown insane (that we know about). So this nest of Warden's losing their minds... nah. Peace.
EDIT: In DA:O the wardens were exiled from Ferelden and were recovering street cred. Exiling them from Orelais just seems par for the course if they can't stop being morons. Can you imagine because Villager Peon Number 3 and living in the village near Adamant? "Oh those Grey Wardens will keep us safe! They fight Dark Spawn and champion the little peopl.... why is there smoke rising in Villager Peon Number 2's house. Are those Grey Wardens with demons following them around? MILDRED GRAB THE CAT AND THE KIDS AND RUN! Maybe the mages will keep us safe."
The moral of the story is that damn near every institution in Thedas is pretty unreliable... except the brothels.