I can see what you're saying there. Just in general that always seems to be what happens. But a lot of times, even having just a little bit of feelings for another person when you are in a relationship plants the tiny seed of doubt. You hope that it doesn't get rained on and grow.
Sometimes it doesn't, other times it does. Why it does for sure is impossible to really say for sure. Peoples emotions and feelings are ever changing and uncertain. So don't blame it on yourself.