If you rivalmance though, it would seem you can't be nice to him at all until you max rivalry (choosing either aggressive or joking responses)… I prefer to just max friendship by the end of Act 2, and then it doesn't matter what I disagree with him on in Act 3. I never let him kill his sister and he still is my best buddy with max friendship. I have a hardtime thinking a positive romantic relationship can start with nothing but disagreeing with someone and trying to change them and challenge their beliefs. I'd say he has a right to be pissed at Danarius and I don't really feel like pushing him to believe his problems are all his fault. He shouldn't blame magic as much as he does, but he had his reasons, and the majority of mages met in DA2 seem to fit with his fears (I don't usually play a mage, & off the top of my head the only mostly "good" mages I can think of are Bethany, Feynriel, and Alain). Maybe, I might try Fenris rivalmance next playthough though… maybe. Just to see what happens. Who knows, might be okay, more so perhaps if I choose to play a mage next game.
I'm not sure how much use there is in responding, because I feel it will boil down to repeating myself, but here goes. Like I said before, aggressive dialogue options are the fastest way to max Fenris' rivalry (and that of all other companions) but that does not mean you are in any way obligated to choose these options if you want to rival him. I maxed his rivalry with a sarcastic Hawke who would joke with Fenris and responded nice/diplomatically when he was distressed (e.g. post Bitter Pill). You spared his sister on friendship? Great, and I never sided with slavers and gave Orana a job, which meant I had to deal with some additional friendship points. It is more difficult to max the relationship this way, but it is doable.
You mention the sarcastic responses as the other option to be mean to Fenris and max rivalry, but actually Fenris - unlike Anders, who really needs all the friendly responses to be able to like you - can handle Hawke's jokes and tends to react positively to them. The only sarcastic option that yields rivalry (+5) that I can recall is when he tells you about his sister's arrival ("yeah, everything is going exactly to plan, how awful" or something like that), and I consider that reaction quite justifiable for a Hawke who is hurt by Fenris waiting this long to tell them he has decided to search for his sister.
Know what Fenris also doesn't like? (Extreme) sympathy. During the conversation after recruitment he tells you about having been on the run for a long time. You can be a douche and tell him he whines too much (+10 rivalry) or you can be sympathetic and say that it sounds like a tough situation. + 5 rivalry! "I'm not made of glass". Pfft. My point is that you can gain a lot of rivalry points from magic-related issues and most of the extra points you need by going for options like described above that yield a small amount of rivalry rather than friendship. Also, flirt options net +10 rivalry if you're on the rivalry path, and they are still friendly.
It's not about trying to change everything about Fenris and everything that he believes. It's about challenging his more destructive tendencies, the ones that actually make his suffering worse. I explicitly stated that he does not start "victim-blaming", i.e. he does not start to believe that what Danarius did to him is his own fault. You can even say to him "Danarius isn't exactly blameless, you know" when Fenris states "what has been done to me, I have done to myself" in Act 3 QB. Fenris' reply is: "and he is dead. There is no one left to blame." And THAT is the point. He has finally killed his former master and gotten his revenge, but now he discovers that it does not suddenly make him happy and solve all his problems. He has to face that though he is now truly, finally free, he is still living in a squatted mansion with hardly any friends and no clue what to do next. He was afraid to build a life of his own as long as the possibility existed Danarius would show up, but that means he has nothing once Danarius is finally dead. The only thing he worked for was find his family (and past) but Varania betrayed him and the only thing he can potentially learn about his past is that he "won" the markings and earned his family's freedom. And then he has to admit that he has played a part in why he is living the way he is, practically a ghost as Varric calls it. Keeping people at a distance, freaking out and running away when his memories return after the night with Hawke and refusing to speak of it and face his own feelings for three years, that's him. Of course his life would have been better if he had never been Danarius' slave and he would not have had the issues he has if that was the case, but the way he (tries to) deal(s) with them is him, and as a free man he has to accept the responsibility he has for his own life and happiness.
Oh my, why can't I ever write short posts? 