First, may I call your attention to the guidelines for using the comments feature found here:
https://www.askmehelpdesk.com/feedba...ure-24951.html
You have inappropriately and unfairly given negative comments to two respected members of this site. When you first use a site like this it pays to browse around and see how the site works and how people use it before you blunder around and use it incorrectly. You made one blunder by piggybacking, another by not reading information that would have answered your question and now a third by using the comments inappropriately.
One of the problems here is that you did not give us some pertinent info. It was unclear whether these children were fathered during your marriage or prior to it. Had we known that, I think it might have changed our responses somewhat.
But I still maintain that there being two children matters here. You stated: "she never gave him the opportunity to have a say so in the matter of her having them since she told him she was protecting herself and on birth control." So why did he believe her after the first child? He's an adult, he should be aware that sexual intercourse can lead to children even if birth control is used.
I do applaud you for standing by him, but the fact remains that he cheated on you for a period of more than 2 years. I have to question how important a marriage is a man when he does that. Also, he must have know about the first child, but he seems to have ignored the possibility that he fathered that child. Sorry but you don't present your husband in a favorable light based on what you have posted. You also seem to have a funny idea what marriage is. It's not philandering, nor is it giving ultimatums. Of course this is just my opinion and you can take it or leave it as you wish.
But that doesn't change the law here. The law will not terminate his rights, the law WILL require that he support his children, at least financially. The law will not require that he maintain contact with those children. So you need to deal with those legal facts. You can't give him an ultimatum to terminate his rights or end your marriage since he has no control over that. You can force him to not have contact, however.