How come that the oxygen on a ketone group can have two hydrogen bonds while an ester group with two oxygen will also have two hydrogen bonds? Wouldn't it be four hydrogen bonds since there are two oxygen atoms?

The formula for Acetone ( AKA Dimethyl Ketone) has no hydrogen bonding C=O. Same for esters. Look at the formula for Methyl Acetate, same C=O linkage.
If I remember my Organic Chemistry, 60 years ago, the carbon atom in carbonyl group (C=O), ketones and esters, is not electronegative enough to give hydrogen bonding.