I understand what you are going through.
But if, just if you get married with him. I bet you, that you will be living with guilt the rest of your life. You won't be entirely happy. Guilt and thoughts about guilt will kill you and lead you into something worse. Don't be a homewrecker . No one loves a homewrecker.
Memories are forever, but will be a burden in this case.