My son's father has not been in contact with us for 6 years. Can I move out of the state without worring about legal action on his end? We were never married and there is not a court order for custody. Is the 6 years considered child abandonment and I have all legal rights?

No, there is no automatic termination of rights because of abandonment. However, since there is no orders of custody, you are free to move. However, if the father went to court before you could establish legal residence elsewhere he might force you to return.

I think that unlikely given the history.