What was so great about rivalry? It just gave you another extreme to aim for, and one that made less sense than approval. You could suck up or be a jerk now, and being a jerk somehow made them just as likely to remain with your party as friendship (the new approval) did. All it did was removing the negative consequences for being a jerk and replacing it with a positive one.
It almost forced you to either kiss their rears or being consistently mean so you'd max out one of the two, because in between was wishy-washy-land, which gave you a disadvantage.
I think you're being a little too general in your description of rivalry.
In my opinion, the states of "friendship" and "rivalry" actually implied a positive or negative outlook to their agendas/lifestyles respectively, as opposed to an indication of whether you were their friend or not.
It's important to note that being a jerk to someone in-game does not always imply being mean. You can call Isabela out on her nonchalance in a serious way using diplomatic or direct dialogue options, but that doesn't mean you were doing it to ****** her off. You're actually concerned for her well-being.
Merrill is the best example to use here.
You gain rivalry points by being against her goal of restoring the mirror, whether you're friendly, cynical, or aggressive about it. In one scenario, being nice about it nets you the biggest gain, a whopping 25 points, when you tell her that you're looking out for her best interests.
Your entire friendship rating ( - 15 points ) is converted to rivalry should you deny her the tool. This implies that the more she trusts you, the more she feels betrayed by your act.
They have their agendas. That friendship/rivalry bar is simply a representation of how aligned you are with their interests as opposed to how friendly you are with them.
Now you can ask why they would stick around with Hawke when they're against their agendas. The game goes through great lengths to make the reasons obvious.
Of course this is contingent on you always being around to help them should they ask you to, irrespective of their agendas. Should you not complete their arcs, they will inevitably turn on you (except for Varric) based on your decisions at certain critical points in the game.
Anders: Needs the protection of Hawke, Aveline, and Varric. Hawke appeals to Anders' human side and keeps his desires in check. The human side of Anders understands this.
Isabela: Needs your help in finding the artifact. She grows to respect and admire Hawke's selfless nature. which ultimately makes her return to you during her crisis point because she felt guilty and wanted to do the right thing.
Merrill: Needs the protection of Hawke and Varric. Hawke continues to be on Merrill's side throughout her fumbling and she eventually understands that she needs to let go of her past.
Sebastian: Hawke is one of Sebastian's allies in Kirkwall. Hawke convinces Sebastian that he should become king.
Aveline: Life debt by saving her ass in Lothering. She's been through you through thick and thin and she considers you family. She only disowns you should you not complete her arc and not get her married to Donnic. That's some serious sabotaging.
Fenris: Respects you regardless of your stance on magic. He owes you a debt of gratitude for risking your neck to keep him safe when you didn't have to. He has bigger fish to fry, so as long as you're helping him keep Danarius at bay, he'll stay by your side since he doesn't want to leave Kirkwall just yet.
That you gain bonuses by being against their agendas is also explained in the passive descriptions. Most of the rivalry bonuses are defensive in nature while the friendship ones are symbiotic/offensive. This makes sense. They have to more on guard when they have to be more independent and can't rely on Hawke's support should you be their rival etc..
So ultimately you being a complete and utter jerk to them by not even helping them at all makes this entire discussion moot since they end up just leaving you. Rivalry/Friendship has no bearing here.