I think a big part of it is society in general.
We make the older man into someone more experienced, with more money, and generally more maturity.
We make the ideal age/look for a woman to be about 18 and skinny... which, try as I might, I can't seem to do at 32.
I think it's that each goes for what looks good on their arm as much as anything else. Some people feel better about themselves if others envy who they are with.