Do I worry>?

That's your call. Chances are your baby will be fine, but watch him/her. If he/she develops a fever, or any other signs that this is making him/her sick, then off to the doctor you go.

Is this your dog, or a strangers dog? Also, how the heck did a baby gain access to dog poo? Who's watching this child? Was this is in a home? If so, who's cleaning? How old is the dog? Why is the dog pooing where a baby can gain access? Those are the bigger concerns.