Let me see, they see the son, using up your ex's visit time. You complain because they are not using up your time?

I would say you are overthinking it, and really does it matter ?