You don't refer to women as she and men as he, though. You refer to people with vaginas as she and people with penises as he. That's the difference.
I'm old, but in 99.9% of cases, people with XX chromosomes do, in fact, have a vagina, and would be referred to as "female; that is, having female reproductive organs, and would be accurately referred to as she. And vice versa for men.
My "shes" and "hes" aren't about some nebulous concept of what people feel inside, but about female/male, which, in my world are clearly defined concepts with a factual basis,
Now, for the other .1% of people, maybe people with something like Klinefelter's, or fragile X, or something like that, I agree it's open to interpretation.
Now, if someone vehemently wants me to address him or her by a certain pronoun, I would do so, in order not to be rude, in the same way if I met someone called "Bob" and he insisted I call him "Slugger", I would.
But otherwise, I'm not wrong in addressing someone as male or female, if in fact, scientifically, that's what they are.