We're having a New Year's Eve party that's starting at 9:30 PM and do not want
Guests to bring small children. We don't want over-tired children who are past their bedtime to run around and cause problems.

What's the best way to tell guests to leave their children home?

Specify NO CHILDREN.

How about "PLEASE, no one under drinking age!"
If they don't know all the hoopla these days about the host's liability, not to mention their own when driving minors after drinking, then heaven help them.

If several guests have small children, then arrange a group babysitter at a nearby home.