If the father has sole physical custody, and it appears that he does, it follows that the mother has no visitation rights which would be impaired. So the only problem is getting a passport. I would check with the passport agency about getting the passport without the mother's consent. As I recall, there are some exceptions to the requirement that both parents sign the passport application.
A form
DS-3053 can be submitted, with a showing of the "special circumstances" under which the non-custodial parent has refused to agree.