My nephew is a US citizen and had a baby with an illegal immigrant. They live together but is a very dysfunctional relationship; she has an explosive temper, extreme jealousy, is verbal and physical abusive to my nephew and is constantly threaten him with taking the baby to Mexico, so He'll never see his daughter again. He stays in the relationship because he loves his daughter and fears that the mother either takes away or mistreats her when he's not around. Could He get for custody of the child or establish visitation if he decides to leave her, even if they aren't married? What rights does my nephew have as a US citizen and a father? What could happen to the mother (legally)?

First marriage has nothing to do with child custody or child visiation,

If they separate they need to go to court and get a child custody agreement, and child visittion agreement, and of course child support payments arranged.

If she did take the baby to mexico, it is doubtful if they would honor any US child custody agreement

And the term anchor baby is only a term giving the mother the right to stay in the county since the baby is a citizen