No matter what he will always have contact with this woman. They will soon have a child together. He will be responsible for this child, he will have to see not only the child, but the mother of the child.

Personally I don't see this as a real marriage. When times get tough, or someone else catches your eye, you both leave to pursue what you want, without thinking about the vows you took.

This marriage will end sooner or later, because neither one of you are committed to it. It seems to be too late to commit now.