When I enter my house, there is a very strong musty odor that I can smell. Although there is no mold visible anywhere around the house, I'm suspecting that we have a mold problem. Any suggestions?

Mold likes moisture and darkness so start in your basement, especially near plumbing. Also check under every sink and around toilets and showers. Look for any sign of leaking or moisture. If you have a humidifier I'd suggest taking it apart and cleaning it, don't reuse the pad.

I did all that you asked, but I couldn't find any evidence of mold. How exactly would an inspector check for mold, especially if there aren't any visible signs? And how much would it cost?

Mold inpection, the kind you pay for, involves taking smears and culturing them. I just look for black specks in a cluster and assume its mold. I spray the heck out of it with bleach but stuff like drywall needs to be removed. Do you have a crawl or a basement and is it ventilated?