My husband and I have had guardianship of our 2 grandchildren since they were babies. The birth father has not seen either child in over 4 years and has never given any financial support. The children's birth mother, our daughter, lives with us.

Do we have grounds to pursue terminating his parental rights?

What custody rights ? He may have right to go to court and ask for visits, ( a visit is not custody)

Does he have long term prison sentence ? Proved drug abuse ( evidence for court )

Just not visiting the child or paying is seldom enough to take away a parents rights.
Next has the mother and the father been ordered by the court to pay support, if you have guardianship the bio parents BOTH should be ordered to pay support. Is there court orders for that. If no, why is there not, he has limited liability to pay anythint unless there is a court order. If there is, has the mother ( your daughter) been paying you support payments also ?