The power in my house goes out and I have to go out to the circuit breaker. Most of the time I'm at work and evrything in the house is off so the circuit breaker isn't overloaded. Why does this happen? Is there a short somewhere ? Electrical problems put the house at risk of fire and that's what I'm afraid of.

What type of breaker is this, is it an arc fault or GFCI by any chance? How many breakers do you have and where do you live?

From the sound of it I would call in an electrician as soon as possible.