In the state of Texas can a father relinquish his parental rights to an unborn child if he is not married to the mother

First no, you can not relinuish or even establish custody of a unborn child. But the father can not relinquish his rights anyway so it does not really matter, And if for some reason they can get them, they will always have to pay child support.

So after the child is born, it will be the mother who will demand a DNA test to prove he is the father and it will be then that the man will be ordereed to pay child support, The rights are easy, there is no reason you have to use any of those rights, just never visit the child, never see the child and that is easy, no need to ever use the rights