If it doesn't feel right, it doesn't feel right. No need to make excuses for him or yourself.
And yes it is his fault that he's a loser. He's in control of his actions and his behavior. No one else. So if he is as you say a loser, it's by his doing no one else.
As far as telling him... be honest. Tell him you feel he has lied to you and that's unacceptable. Done.