Well what kind of situation are you in and who told you to do this?

We were never married. My son's always lived with me. He's 6 years old. I am in a new relationship with a man who lives an hour away in the next state. We spend the weekend with him and his kids sometimes. My son's father isn't happy with this and has told me that legally I need to provide him with the address of where we sleep if not at home. We do not have any court ordered custody or child support. Is he correct?