What can I do to stop this? I don't know why it happens and usually goes away if I leave it alone

Sounds like it could be seborrheic eczema or seborrheic psoriasis. Here is a link that might help you figure out what problem you are having:

Adult Seborrheic Eczema (http://www.internationaleczema-psoriasisfoundation.org/adult_seborrheic_eczema.php4)