My ex and I divorced when my daughter was 2 due to his drug use and sexual relations with a 16 year old girl when he was 38. The courts made me the primary legal caregiver but awarded joint custody. He would see her sporadically - about 6 times in one year and then he moved away when she was five. We now haven't seen him since aside from one 4 hour visit almost 2 years ago. He has made no effort to see her or call her. Can his parental rights be terminated based on abandonement? He sometimes pays support but it has now stopped again.

Contact an attorney and file for sole custody. If he does want to see the children that would be his choice, as for abandonement he did, and that will be your case for sole custody, also have child support enforced.

Strictly speaking, the only way his paternal rights would be abandoned would be if someone else wanted to adopt your daughter. Normally, the father would have to give consent so you'd have to demonstrate that you've made every reasonable good faith attempt to contact him. If he's then nowhere to be found, the adoption could proceed without him and his rights would be terminated.

“...Defined
A termination of parental rights means that the person who was the natural parent of a child no longer has any rights or responsibilities to that child.
Rights:
Rights regarding a child include the right to decide what kind of education, health care, religion, morals and values the child should have. Custody rights and visitation rights are also associated with children.
Responsibilities:
Responsibilities include the duty to provide food, clothing and shelter for the child, provide all necessary child support, daycare, etc.
A parent whose rights have been terminated has the same rights and responsibilities toward that child as a complete stranger. Such a parent is not responsible for any support, nor is that parent allowed to have any input or influence over the education, teaching and upbringing of that child. In fact, a parent whose parental rights have been terminated does not even have the right to see or contact the child.
There are two ways by which parental rights may be terminated.
Voluntarily
The natural parents may voluntarily consent to the termination of their parental rights, such as when an adoption is being permitted, and the child will live with new adoptive parents. Court approval is required for this kind of proceeding.
Involuntarily
Parental rights may also be terminated involuntarily. To terminate these rights involuntarily, the moving party must demonstrate that the natural parent or parents have abandoned the child.
Abandonment is demonstrated by showing that the parent has, by conduct continuing for a period of at least six months, either evidenced a settled purpose for relinquishing parental claim to a child, or has refused or failed to perform parental duties. A petition to terminate parental rights may be filed by either parent, an agency supervised by the Department of Public Welfare and providing adoption services, or an individual having custody or standing in loco parentis to the child.
Note: As a general rule, courts are reluctant to terminate parental rights when one parent feels that another parent is unfit. Even if one parent has lots of flaws, courts are hesitant to simply relieve a parent of his or her duties to properly raise the child and pay for the child's expenses.
It is extremely difficult to terminate parental rights, and courts will do so only in rare circumstances...”

http://www.familylawyerservice.com/termination.htm

This law is based on information collected from a PA. law site but I also check CA. and Va. The same information is available... it is extremely difficult to terminate parental rights and the judge makes the final decision in all cases...