If a father signs his legal rights away to a child when he was 17 years old, can he regain custody of the child?

NO but by 18 you should be able to have a relationship with him paper or no paper as long as he is open to it. By 18 he is considered an adult and can choose to be with you unless there is a court order stating different.

You mean when the father was 17 or the son? Courts do not take a Termination of Parental Rights lightly. So if one was granted, its unlikely it can be reversed. However, if he was a minor when he relinquished, there may be a small possibility of it being reversed. The father NEEDS to consult a Family Law atty