Could it be that, Your brother and Fiancé, only ask to take your son or have their son spend time with your son when your ex has him because, They respect and care for you and your relationship time...