My ex uses drugs when he has our children every other weekend for visitation. My kids don't like that he does them. Can I not allow them to see him if he continues to do drugs while they are in his care? We do not have a court custody agreement. We had a verbal agreement made and the kids live with me and visit their dad alternate weekends.