I had similar issues with Envy, but manged it after a few tries. It was definitely the hardest part of the game for me so far.
I don't know where to start, but essentially, the game seems to be not designed as well as the first two combat wise. Barrier is the only really good way to mitigate incoming damage and melee is punished for being melee due to AoE's.
Envy was just ton of hp and unblockable aoe's with cc immunity. That's not really a fun combination but it occurs quite often in the game. I had two warriors for this fight at they were the constant target of those AoE's. The best strategy for first and 3rd phase of the fight (when it uses the pop up aoe) I found is to control the mage and use barrier on whoever is about to get AoE'd. If barrier is on cooldown, well you better have potions left. If I'm ever in that fight again I would bring 2 mages and I know it would make the fight a lot easier.
This is a pretty annoying design decision though. Melee gets punished heavily for being melee like this game is an MMO or something. There are so many AoE spamming CC immune enemies that having more than 1 melee (and that 1 melee better be a shield tank since shield wall is best survival tool in the game) is really asking for your life to be difficult. This wasn't a problem in the first game since 360 AoE hitboxes of death weren't that common, it only really came with some bosses, like dragons. But because we are a quasi-action game now pretending to be Dark Souls or something the game is filled with CC immune 360 AoE enemies as if you could dodge/block to overcome them like in an actual action rpg.
This may not be a huge deal on normal, but those kind of situations on nightmare are basically potion sinks if barrier is on cool down. You can technically avoid some of that aoe damage with roll/evade but at the cost of losing time on target. And since movement is sluggish and those abilities don't come with invincibly frames, like Dark Souls, it may not even work. An issue that does no affect ranged classes.
So in short, my advice for Nightmare is to avoid melee classes and/or get 2 mages to get the most out of barrier. Barrier scales very well and having barrier up is akin to having temporary invincibility. Roll/Evade don't even come with invincibility frames, Guard is borderline worthless for damage mitigation since it does not scale, which leaves us with barrier as the only viable method of preventing melee from taking damage while letting them actually function in close combat. Barrier makes these kind of bosses/elties cc immune aoe enemies a lot easier to deal with, so carry 2 mages around.
They really should fix those hitboxes as well, it would make melee a lot more viable. Hate seeing my rouge get hit attacks that clearly are nowhere near him because the hitboxes are so messed up.