Its been four years since the father of my child has made any contact with me and has never once in these four yrs tried to see his kid is there a way to get him with abandament or some how have its rights taken away with out my current husband having to adopt

The short answer: no

However, he isn't using his rights, so why would you?