IMO, the best party setup for Nightmare is 3 dagger rogues and a rift mage. Solas has plentiful crowd control and rarely runs out of mana, so he can lock down enemies while the dagger rogues gut them. The dagger rogues have enough DPS and evasion (through Stealth, Evade, upgraded Flank Attack, Hook & Tackle, etc) that their squishiness doesn't matter. The enemies are dead before they lift a finger. High dragons die in a matter of seconds with a 3-rogue party, even if the rogues are unoptimized.
That's sound advice - but it's an entirely different game. Instead of finding a way to beat the enemy, you're figuring out a way to deal XXX amount of damage before the enemy gets to do anything. It's a math game, rather than an action/tactic game.
By design a team of DW rogues can never be challenged, because you simply can't beat content that's too hard for your team, and content that you can beat, well, never gets to do anything to challenge you at any point.
That's not the case of playing more conservatively, more defensively, where the difficulty of the content transcribes exactly into gameplay challenge.
I don't know if I make sense.
Things such as soloing Gurd. People say Assassins can do it in under a minute, but so far we have only one testimony of a person who did it in an hour with a two-handed Templar. Is it because Assassins can't get to Gurd? Or is it because a solo Assassin can't mathematically kill a boss before the boss gets to guard up, freeze the assassin or summon archers and more mobs? Or is it because a solo Assassin can't mathematically beat one of the fights on the way there on Nightmare? Or was a video done and posted already, and I missed it?
In any case I've yet to see a full-on damage oriented approach to the more challenging solo/duo content available, and yet I completely agree with you that 3 rogues and a mage is the best nightmare party, performance-wise.
My conclusion to that is also meta-gamey: Nightmare still isn't hard enough to warrant more defensive play with 4 party members. Players like me consciously make it harder because we have more fun when it's more challenging. So I will purposefully pick my teammates for roleplaying reasons over performance, but I also won't hesitate to gimp them or even gimp my own character to make every single minute played more challenging, until I can't gimp myself anymore. Then and only then I will suck it up and accept to buff myself.
We won't be satisfied until we have content that makes us ragequit.
That's why in my party Vivienne, like all other mages, unequips her staff, has an empty skillbar and is a glorified veilfire torch holder and energize bot. Don't take it the wrong way, I also consider Cole and his rogue friends as walking lockpicks.