A friend of mine recently came out to her parents as being a gay woman.
Her mother is telling her she plans to take my friends daughter.

What would deem a parent unfit in the state of Texas?
Could my friend's homosexuality be held against her in the state of Texas?

Unfit isn't a law, it's a determination based on evidence by a court. Homosexuality is not a determining factor and if its mentioned by the plaintiff the judge might just dismiss the case.

To be found unfit, one has to be ruled a danger to the child, with evidence showing neglect, abuse or other detrimental factors such as prolonged incarceration.

Or in other words... it isn't a crime to be gay.