I know this sux to hear, but there's a possibility he's cheating on you . If you had sex before, what difference does it make now?
Best of luck

Now that more info has been provided, something does seem a little less innocent about the whole situation. You do need to have a serious talk with him about what is going on but it's important to remember to confront him in a non-threatening way so that he will be more apt to come clean about his intentions. Good luck.