I just found out my ex-husband was giving our boys ages 4 and 6 Benadryl every weekend night. Is that child abuse. They have no allergies.

Have you any idea why he is giving it to them? Itching, perhaps?
I wouldn't leap right into the inflammatory words 'child abuse' yet, but regular (ingested) Benadryl is not for anyone under 6. Have a talk with their doctor first. Presumably you and your ex take the children to the same doctor?