My dog recently had a litter of 4 puppies, 3 males and 1 female. They are 9 weeks old now and I noticed that two of the males have what seems to be a pocket of air or fluid on their belly button. When I push on it; it doesn't seem to be painful to them at all.
I really just want to know if this is something to worry about, or will it go away on it's own eventually? Does this bother my puppies at all, and is it painful for them?

It sounds like a hernia, the most common cause is from being a little too rough on the umbilical cord during cutting.
However they can just happen randomly as well.

Depending on the type of hernia they may need surgery. What did your vet say when you took them in for their vaccinations?

Hernias can be fixed when the pups get fixed. :)