My ex-spouse won the right to claim the children on taxes, but he is delinquent of over 2400.00 in child support. Can he still get those tax exemptions and/or credits

Yes, of course, being in default of one issue of a support or custody order does not take away any of the other provisions. You can see if the court will allow you to attach or garnish his income tax refund for his back support.

Again, non payment of support does not take away from other rights such as visitation, tax benefits and others.