We don't really know how powerful the Inquisitor's mark is, but the smart money's on really terrifyingly powerful.
We keep learning uber-powerful skills for it, and plus there's the small matter of allowing its wielder to walk physically in the Fade. And also control elven orbs, which are demonstrably super dangerous.
If the Inquisitor wasn't me, I'd want to get rid of them too. I wouldn't know their values or morality (actually, even knowing my Adaar's morality, I wouldn't trust him with the mark). For all everyone else knows, the inquisitor could be just as corrupt as Corypheus. I don't really blame the Qunari for wanting to eliminate the inquisitor, if that's their motivation. The inquisitor has way more power than anyone can safely have.
It's a small point, but I've always been curious whether Abelas' comment about the anchor was meant to parse like:
"You bear the [mark of magic] which is familiar."
or
"You bear the mark of [magic which is familiar]."
In other words, is the anchor itself something the ancient elves were familiar with? Or did he just mean rift magic, in general? Subtle distinction, but it might mean the difference between Solas knowing exactly what's going on with the Inquisitor's hand, and not.