We have a case in Texas where the father wants to sign away his rights to his children as he is mentally incapable of caring for them. Their mother passed away Oct. 6 and they are 1 and 4 years old.

It may or may not be allowed. What is going to happen with the children? Are they going to be adopted or are they entering a form of foster care? It may not be so easy to sever the rights and responsibilities of the father.

There is no "form" normally unless he is signing rights away to allow an adoption, an attorney handling the adoption will write up the request.
A judge may or may not allow it.

Then this leads to a serous question, if he is not mentally capable of caring for the children, he may not be mentally capable of signing paper work, in which case he would have to be declared non competent by the court, have someone appointed and so on.

Who is wanting him to sign his rights away? And why.

It is almost impossible to sign rights away, outside of adoption,

But there are no "forms".

Something does not sound right about this. If a court ruled that the father was incapable of taking care of the children then the court should have, as part of the same ruling, awarded custody of the children to someone else. Or allowed for the adoption of the children.

There are no forms here. Only a court can terminate parental rights. So the court should have issued an order about custody of the children.

So you need to explain more of the back ground here. Like who is "we" (I'm guessing by your screen name that you are the maternal grandmother)? What actually happened here. What happened when the mother died. Etc.