I think that Merrill's rivalry is more about opening her eyes to the consequences of her action (The fact that fixing the mirror cost her her clan, as they are either dead or will kill her if she returns to them). same as with other companions. As well as friendship is helping her with the zeroing in on this one issue, as if it would fix all her problems getting the mirror working.
No, I've actually watched quite a few rivalmance videos with all rivalmance options explored. The rivalry path is about being massively cruel to Merrill at every turn (emotional abuse at its finest), and/or constantly trying to undermine and sabotage her attempts to fix the mirror long before the "consequences" (her people trying to undermine and sabotage her) ever transpire. Figuring that Merrill is messing with powers beyond her understanding even though Hawke knows little to nothing about the eluvian, but Hawke must be right because s/he's able to see things "objectively" while Merrill is too emotionally invested and unable to see things clearly because she's too close to her own culture, does not make Hawke right in my book.
Nothing says an emotionally healthy relationship built on trust, respect, communication, and common values quite like not trusting that Merrill knows what she's doing (what she's researched and studied extensively on), not respecting her desire to help her people and fix an artifact that can accomplish it nor respecting her right to make decisions like an adult, not communicating so much as telling her she's wrong and should stop at every turn and even just flat out refuse to give her a tool she needs at one point, and telling her that her core values in helping her people, restoring parts her people's history, and reviving an artifact that can help accomplish that are wrong, she should stop, she should focus on something else, etc.