Ok my husband and I have been together for 6 years and his family is always calling me his x girl friend? I know have 2 kids and can't cope with it any more. Does it make me a bad person if I stay away from them?

Ok so my husband and I have been together for 6 years and have 2 children. He has a child from a previous relationship and has nothing to do with this child simply because the mother made it so he can't be her dad every time he tried she always finds away to have him put in prison. His family like his mom and grandma etc always call me his babies mom and they do it all the time and in front of my children and I can't cope with it anymore! I want to just stay away from them but my husband tells me I'm being selfish and te kids need these people in there life? What do I do before I have to look into divorce over it?