Their reaction was "then why do you feel compelled to continue using the word?"
I didn't really have a good answer. Years later, I feel I felt defensive because I knew being a racist was a bad thing, and if I admitted to using a racist slur then my personal self image now has to reconcile the question: "If I acknowledge that I used a racist slur... does that make me a racist?" Which is a terrifying thing to face because I knew being a racist was wrong and I didn't believe I was a racist. The brain (which I do not believe we have full control over) is exceptionally good at rationalizing things to eliminate cognitive dissonance, and it's a major amount of cognitive dissonance to try to reconcile the idea that I performed an action that I would typically classify as only being done by racist people - a status I didn't want to believe I could be.
I can see your line of thought, though I myself say not, using a racist slur, especially out of mere ignorance, doesn't make on a racist. Racism is an act of believing that there are inherent differences and traits specific to a race or culture that allows it and its members to be categorized as inferior. Without that belief, then you can't be a racist. Furthermore, I would say that while the above definition makes for a good academic definition, in practice racism to most people is acting upon those held beliefs to discriminate against a set of people for arbitrary reasons. Just using the word while lacking the belief, the intent, and any other ill feelings, means that it isn't a case of racism. Ignorance? maybe, but not racism. I would say it would take more mental gymnastics to come to the conclusion that a word in it of itself makes someone a racist, as it flies in the face of racism actual definition and the more popularly held definition that I myself have seen people use when determining what is or is not racist.
I suppose they have the choice to not feel offended. The thing is, if this is true, then I have the choice to not be bothered if their reaction is anything more than allowing bygones to be bygones. If they can choose to not be bothered by that, I can choose to not be bothered by them demanding apologies, and I can choose to not be bothered by them "lording my misunderstanding over me like it was some kind of scarlet letter."
Yes, it's what I've been trying to say. A system where both people involved are not held to some compulsory standard of behavior and acknowledge that fact. They can say their offended, I can say I'm offended, and both of us can choose to not act upon each others mutual feelings of being offended. And if a person does change, its because they are making a choice to be courteous and not being forced to do so because of some unspoken rule to do so, or you're instantly a racist/sexist/bigot/whateve word people want to use depending on the act.
Because sometimes it is a personal thing to not want to be courteous to someone, not what religion, nationality, culture, race, or any other factor that might compose their being, but just that singular individual is the one you don't want to show courtesy to. For instance I doubt stephen fry would very much want to be courteous to Bill Donohue, leader of the US catholic league and anti-LGBT advocate. And it should be his right to be uncourteous to him, donohue's right to say he's offended, and fry's right to not have to make ammends if they don't want to, to use an extreme example.
In practice, I am sure that a lot of cases are like yours, and a lot of them are resolved the way they are because we are all generally good people. I just want peoples ability to be jerks from time to time not taken away, because while I don't always or mostly agree with those views, I will defend their ability to express them, even if the consequences of saying those could be bad. Just no mob justice would be nice.
I feel Stephen Fry's quote is misused a lot. His quote is specifically with regards to a law that would prevent negative commentary to "stir up religious hatred." It's important to note that Fry's views don't really align with organized religion. They use it as a general position "no one can claim to feel offended" and to rationalize continuing to act like a jerk.
It first comes up in an article by The Guardian. It should be noted that Fry takes offense to things like homophobia, antisemitism, and racism. Like most people, he probably is disinclined to validate "I am offended by that" for people that espouse morals and values that are not in alignment with his own.
I don't know what his response to insisting on referring to a trans person as "it" would be (someone could ask on twitter I suppose), but I actually wouldn't be surprised if he felt that someone insisting on using the term was a dolt.
I am hoping he had the understanding to know that what he said had universal application, and wouldn't try to monopolize that line of reasoning or excellent piece of profound thought for the use of only one side of one issue. That would be quite boring and hypocritical, to not acknowledge that perhaps being offended really isn't something that holds as much weight as people have made it into.
Regardless, thank you for your thought out and well written responses. I enjoyed reading them and making my less thought out and shorter responses to those responses.