I was awarded Sole Legal and Sole Physical with contact solely at my discresion. If my kids do not ask to talk to their father do I have to let him talk to them if HE calls? He is only allowed supervised visits at my discresion as well.

Is there a reason he doesn't have a visitation schedule set by the court?

Clearly any contact is at your discretion. But without knowing more we can't tell whether you should.

How old are the children? Why did he get only supervised visits? I would not be inclined to come between your children and their father unless they don't want the contact and are mature enough to make that decision.