Interesting sexuality question, stemming from erectile dysfunction (23y)

Hey guys!

First things first, I'm a 22 year old, fit, healthy male, and I do not have a porn addiction (only masturbate to porn 3 or 4 times a month). I have no issue getting an erection when i'm by myself, and generally get morning wood.

For as long as I can remember I've always found females attractive and I never had an interest in men. Up until a few years ago I would constantly get an erection just from a kiss, or any such contact with a woman. That was until sex came onto the table when I was 20. When there was just foreplay, but no chance of sex, I could still achieve an erection, but if we were in a place were sex was possible... no dice. Eventually I attained erection and lost my virginity, and since then I have had many further attempts, to the point that I have now had sex (if you could call it that) with 14 women. It has gotten progressively harder to achieve erection from any amount of contact with a woman, and whenever I have attained an erection, maintaining it is almost always impossible, so I have only climaxed with 3 women (first 3). After the continued fails during my last attempted relationship, I have given up on dating women for the timebeing, as I can't handle the embarrassment of it any more.

Over the past year and a bit I have had 2 incidents with men (who I have never found physically attractive), and my performance was only slightly better, though I was unable to maintain my erection. Throughout my youth I never had a gay thought, and even now when I walk down the beach its still the women that catch my eye. I haven't gone to a doctor because I know I can't be prescribed viagra if I can achieve erection on my own... so what should I do? Is it possible to be sexually gay, but mentally not be physically attracted to men? As in my penis be gay and my head be straight... Seems a bit off, no?
(I'm expecting some "you're a gay in denial" responses, but if that was the case I'd hardly have tried it out to check. I'll accept what I am... I just don't have a breeze what that is at the moment!)