It's an immersion breaking thing.
Think this way:
You and this girl are friends, been friends for a long time, do things together, have a good time. You have ZERO romantic feelings for her. She's like a sister to you. One night she flirts/makes a move on you. Oops. She clearly is into you in a way that you are not into her. That doesn't just turn off. It's not a switch. You are not going to remain friends like you were. Impossible. She's embarrassed and perhaps hurt, still pining away for you. You have no interest.
The end. Works the other way around. I CANNOT be friends with a girl I want but who doesn't want me back. I WANT to make it go romantic, I WANT it to be romantic, she's interested in some other guy. Can't go on being "buddies" because that is ALWAYS sitting there. I cannot just switch it off, no one can. The relationship changes in an instant and cannot recover to what it was.
Doing that with a character in a game where you were otherwise immersed, breaks it. It's not "realistic" to think you can just say "no" and all's well. The bag was dropped right there at your feet and cannot undrop.
It's a matter of writing and making sure choices are clearer BEFORE you get to points you didn't want. It's just a game, sure, but while you're playing it, if it is written and graphic'd well, it takes you into it to some extent. Getting kicked out of it is...not fun. Is disruptive. It's survivable, sure, but it isn't necessary.