I have joint custody and it was my weekend with my son his dad picked him up to watch a football match which I agreed to but his dad called and said my son didn't want to cone back and just kept him , can he do that

Not if it was your weekend. Did he say why the child didn't want to come back ?

You say you have "joint custody". Is there a court order providing for this joint custody? Does it say anything specific about who gets the child when?

If there is a court order, can he do it, not legally, but nothing you can call the police about,
You will have to take him back to court for contempt but it will often take more than one violatoin for court to do anything

You said it was your weekend. That implies there is a court ordered visitation schedule. If so, he can't keep the child when it is your turn. You can go to the police with a copy of the court order and they will come with you to retrieve the child.

If you don't have such court orders, then you will need to go to court.