The Grey Warden's as a group are one of the strongest groups in Thedas and a individual Grey Warden has to be pretty strong to survive their joining but it's not an instant win against any opponent. They can be beaten by superior numbers, bested by an enemy with superior skill or tactics, or just fall prey to sneak attacks. Your average Warden can probably go toe to toe with your above average enemy of the same class but that's not what makes them legendary, they fight against the faceless hoard (darkspawn) and kill immortal gods while draped in mystery and a legend of invincibility. Are they really invincible? No, they're just highly skilled warriors/rouge's/mages with an quasi-immunity to the taint, but the common man doesn't know that and would rather avoid a fight with them than find out. Another thing they have going for them is the fact that they're largely neutral: They fight the darkspawn, that's it, no conquest, no politics, no power struggle, so there's no reason for the vast majority of trained armed forces to fight them...until now. For whatever reason the Grey Wardens are against the Inquisition, the reason's are unknown but the fact is that they're being forced to fight something that they've never really had to do (at least they've not done it well in the past) and that's fight a massive group of free thinking and highly skilled warriors that are as committed to their cause as the Grey Wardens should be to theirs.
Exactly. Grey Wardens are elite and the best to fight darkspawn, helped by their legendary past, but there are more things out there. Not only different enemies to the ones they are used to (darkspawn), but also other elites that, in lore, question the Wardens' right to claim they're the best fighters in Thedas.
-Seekers of Truth: according to lore, they are recruited from the ranks of the best Templars. Examples of badassery in lore: Cassandra in Dawn of the Seeker, Lambert in Asunder. Very important this group, since it's pretty likely that the first soldiers of the Inquisition will come from Cassandra's Seekers.
-Chevaliers: the best of the best of Orlais. According to The Masked Empire, they train to fight in every possible way, and they study Templar and Grey Warden techniques too. Examples of badassery in lore: Ser Michel and Gaspard in TME.
There's no contradiction with the previous lore, since in Ferelden there weren't really any elites to dispute their badassery. But Duncan admits that, since a long time has passed since the last blight, they're in decline. After all, when you don't have many darkspawn to fight, you get rusty (ok, probably the Anderfels Wardens are fearsome fellows, since that country is plagued with darkspawn, and our friends in Amaranthine too).