Why should my ex get to see our son when he neglegts him when he has him?

What do you mean by neglect ?

But he should be able to see him because he is the father

He gets to see him really because there will be a court order for visits that has to be followed.

If you disagree with the terms of visitation, you have the right to go back to court to try and have it modified