He is supposed to see them every other weekend but only ends up seeing them about once every three months if he can be bothered. Although the kids worship him and he winds them and us up by telling them it is us that are stopping the visits.

The only way anyone can try to change a court order is to go back to Family Court. But it's not likely that a court will stop his right to see the children, no matter how seldom he sees them. Plus - think about it - if the kids worship him now, think how much more they will worship him if they can't see him at all!
As for winding you up, you are allowing yourself to be wound up. Don't even talk about his missed visits. They are gone, over, done, in the past, and he knows he can twist it all around by accusing you, and you fall for it. Just walk away and let the kids have their precious time with him, and try not to resent it.

You need to get the children ready when its his time to visit. If he doesn't show up the children will know its not your fault.