First, may I call your attention to the guidelines for using the comments feature found here:
https://www.askmehelpdesk.com/feedba...ure-24951.html
Please learn to use this site properly. We HAVE to make judgments because we rarely get the whole story. True we don't know you, sop we have to go by what you told us AND use our knowledge and experience to make judgments. You owe smoothy an apology!
Second, this is the Family LAW forum, Answers here need to conform to existing law and laws are not always fair. You asked what your rights are. But you forget that he has rights too.
Having your son sleep in the same room is not that big a deal since he's very young. Talking you down to him is not good, but a court will tell him to stop it, not use it to deny him visitation.
Now you said he wants to keep it out of the courts. That may be to your advantage. But the question is whether he's the legal father? Is he on the birth certificate? Because if he is he has equal rights. Therefore, he can keep your son and you would have to be the one to go to court to recover custody.