My son is 5 years old and has spent little time with him since birth. He fathers children throughout the area and claims he has no son. Xmas, birthdays, and other holidays he doesn't contact at all. I want to know if I have legal grounds to sue for abandonment, child neglect, and taking his rights away.

Probably not.

You CAN, however, file for child support and full legal and physical custody.

He's not USING his rights.

Usually the only way a court (not you, a court has to do it) will sever parental rights is if there is a stepparent willing to adopt, or if the child is endangered by the biological parent.