Some of my thoughts so far:
I'm not very far in this game, maybe only like 7-8 hours, but it's very dissappointing so far unfortunately. It's not a bad game exactly, but it's definately not great. It has a lot of problems, mainly with combat. Spider tried to build up the vibe that it wasn't an easy game in interviews, and they're right. It's really not. But the reason for this is that the combat system isn't done very well, not because the combat is challenging. It just feels really sloppy. And that's coming from someone that is genuinely trying to fall in love with this game, because i've been looking forward to it ever since it was announced. You do so little damage, and the enemy can kick your ass in just a couple hits. The two handed style is ridiculously slow, and the damage you do is hardly enough to make the loss of speed worth it. The daggers are slightly better, but still, combat quickly becomes a matter of "when the hell is this fight gonna be over?" instead of "damn this is fun!" I'm not gonna lie, there have been a few moments where I've actually enjoyed the game. But it's not enough to drag it out of the "meh" category.
Sorry, had to go on that little rant lol.
Anyway, back to your little issue I would suggest the following.
If your having problems (and haven't done so already) I would suggest being evil and just going full pyromancer as soon as you get the chance. I don't ordinarily like being evil in my first playthrough of this type of game, but the damage your able to do is pretty insane compared to the other trees when you put a few points into it. Still not enough to make the game a breeze by any means, but I started a second character just to see how pyromancy was and it definately is a lot easier. If you want to be good, then I would focus on the whole daggers sneak sneak stabby gameplay. Even though the damage hit per hit is the weakest of all the trees the speed of the attacks make it the best(in my opinion) of the melee trees. All I've really been doing is hit hit, dodge, hit hit dodge etc. It's not particualy enjoyable or exciting, but it works. I haven't beaten the game yet, probably around halfway(from what i've read in reviews it lasts around 15 hours) in my main playthrough but i'm making progress.