3 main reasons:
1) it's a class based around Crowd Control in a game where you can only control opponents who you don't really need to CC. As cool as it is spamming spells and moving opponents around the battlefield, it's a form of gameplay that works better in theory than in practice. Against anything that can ignore CC (bosses, dragons, big demons etc), the class boils down to spamming Stonefist.
2) From a theme point of view, it can be a little awkward. It's essentially a dumping ground for all the popular spells from the first two games that didn't fit into the new elemental spell system, lashed together with a debuff mechanic. For players who like to theme their builds, it can be a bit difficult to justify what fireballs, chunks of rock and black holes have to do with each other.
3) it's buggy. Stonefist is supposed to work as an impact detonator (letting you pull off your own combos)... It doesn't (the extra damage does not occur). Weakness is supposed to be a debuff that you can keep up permanently... But it cancels itself out quite often.
As Tishen says, it's nonetheless a scholarly class that lets you do some pretty awesome stuff, but it does have a fair number of issues that hit its viability. My opinion is that it's generally a better spec for Solas than the Inquisitor.
What Jaeger said. Say hi to some Kaijus for me. 
The biggest issue with abilities in this game, especially crowd control abilities lies in the fundamental difference between DAI singleplayer and DAI multiplayer.
Now yes we have heard countless times Bioware saying that one does not affect the other but only the naive will believe such a thing.
You see, DAMP was designed to be a dungeon crawler from the outset. Bioware also used DAMP as a platform to test out and to hone abilities for both DAMP and DASP.
On the surface, that might not seem to be bad in anyway for it does result in reduction of development cost. However, one must remember that DASP is a multi-region open world environment whereas DAMP is a strict dungeon crawler.One is about vast open spaces while the other is about tight enclosed spaces.
See the problem ? The two do not mesh well together. As in if you design an ability that fits a dungeon crawler, then it will most likely work in a setting with lots of enemies in a small area.
DASP has low to medium amount of enemies who do not cluster in a small region. They are spread out. Which means crowd control abilities that would work in a dungeon environment will be quite useless in a large semi-open world environment.
You see those Wall of Element abilities ? Horror ? Those things work like a charm in dungeons but they are crap in the open world and they are most certainly crap against many boss creatures such as Giants and Dragons, which do not appear in DAMP. Heck, look at Rift Mage abilities. Most of the time one gets the best out of Rift Mage specialization in a dungeon environment. Watch videos on YouTube if you want. Rift Mage does okay in the open world, excellent in dungeons or in tight chokepoints and crap against large bosses.
As for dealing with mooks, you do not need Rift Mage at all. Basic spells will do just fine and in fact do a better job, since they just require you to gain levels and not do a ques and enemies are not clustered in large groups that often. Energy Barrage with something like Immolate will kill them just fine.