I too have always grown up with ma'am as a term of respect.
Same as 'hon'
I ran into one lady that had deep rooted personal issues to 'hon' and would go batty on people who said it to her regardless of what they meant.
Later I had found out that her husband would beat her and use the word 'hon' in a scathing way.. so rather than facing the person and his behaviour as the cause of her feelings, which she felt helpless about, she connected her feelings of indignity to the word "hon". It became a trigger for her. Perhaps she felt that it was something she could control more than her husband - which is out of her control.
I'd look at how the person said it.. if they are saying words in a mean way, they could make any word sound derogatory. And what would we do then? We can't ban all words that they would use. Why give them so much power over us?
Words alone are just words..
The words a hurtful person would use are just a symptom. A weapon. A utility. It's one of the many ways hurtful people reach out to touch others. If it weren't words, it would be actions. And we can't punish everyone else and what they say or do over what one hurtful person did.
It would be more effective to hold the person, the root of the problem responsible for intent rather than everything they say or do.
I vote that if you do have a person in your life saying simple words like that to you in a derogatory way, you should consider walking away from the person and consider your life without them or at least give them the ultimatum. After all, who needs people like that!