Yet they were both willing to negotiate with the Inquisition, which contained both mages and templars from the get-go?
Almost all of the harsh Templar leadership is wiped or turned into Red Templars out in CotJ even in the BEST circumstances. The remainder get along with mages fairly well.
Yes, both groups contained hardliners. This wouldn't stop the majority of them from changing their positions once they were revealed to be dupes.
Stragglers and singletons aren't institutions. Joining up with an organization that has a few individual mages and templars isn't the same thing as showing up to Haven and finding out that the official faction you oppose and which opposed you has set up camp a hundred feet away. Whatever we see of templars in the Inquisition, they're not having to deal with the same mages who refused to submit to them and broke away from the Chantry to get away from them - they're dealing with a handful that are loyal to either the goals of the Inquisition (Solas and Dorian) or to the Circle/Chantry (Vivienne, and maybe a few unnamed, unseen individuals).
I'll concede that technically it might work if you conscript the mages and then go after the Templars (the bulk of the institution that followed Lucius), but then ultimately you've got the same situation as existed before the Conclave: two factions with diametrically opposed goals. You might be able to make it last long enough to seal the Breach and evacuate Haven, but then you're back where you started: with two warring factions under the roof of a group that I don't think has the political or military power to act as an effective intermediary (not until after WEWH at minimum, with the backing of the Orlesian army). And worse, now the war's literally on your doorstep.
"The enemy could not follow, and with time to doubt, we turn to blame." Can you imagine the clusterfsck it would be to have still angry mages and templars (and their leadership) in the same camp after Haven? Because nothing about their grievances has changed. Their goals (survival and freedom for the mages, control and maybe self-determination for the templars) aren't ones the Inquisition can meet.
ETA: Ultimately, there's a difference between recruiting the faction/insitution and saving a few reasonable stragglers. I wish we'd been able to do the latter, especially since saving a faction certainly didn't reduce the number of Red Templars or Venatori that we encounter in the game. I think it could have worked, even without a second use of the amulet, because neither Fiona nor Barris seemed particularly stupid, just manipulated. But recruiting both factions wasn't ever going to work. They're two groups with mutually exclusive goals.