Can a man sign over his right of his child to the other parent if the other parent agrees and if so is there any other way doing it instead of paying for a bunch of court cost?

No, first even in court most likely the father can not sign over his rights ( unless the mother is remarried and the step parent wants to adopt)

And they can not ever ( NEVER) just sign over rights without it being approved by the judge.

Also ( with the exception of step parent adoption) even if they are allowed in court to give up their rights, they will still owe child support.