The difficulty with balance is that not all players are created equal. An ability may merely be useful when an average player uses it, but in the hands of an incredibly skilled player who uses it optimally it becomes insanely overpowered. So where along the spectrum do you attempt to balance?
This is the reason why difficulty levels exist in a game.
Plus difficulty in using a skill can be a drawback, and you don't exactly have to balance your game around the top 1% of skilled players in a single player RPG.
However in Dragon Age Inquisition for example, I wanted to play a melee mage so I went Knight Enchanter. Unfortunately it was so overpowered I could only die if I chose to, because Spirit Blade could be spammed to have a near infinite barrier. Using the specialization I was most excited for ultimately made the game less fun for me because it wasn't balanced and the most effective way of using it was spamming 1 button.
The same thing happened with my assassin in Skyrim. Bethesda didn't catch multiplicatively stacking modifiers so I can sneak attack with insane damage bonuses to effectively 1 shot almost anything in the game, even on higher difficulties.
I agree about the drawback, but the drawback could be how difficult it is to learn, or how unlikely it is that a character who could learn it would survive that long.
Difficulty of use, or a high damage ability only being accessible by a squishy characters can be drawbacks of it.
If you're gonna make your mages very squishy then they should be doing high damage anyway, as you need a reason to take them over the more durable front line fighters.