I'm so sorry that it worked out like this. Why do you think he freaked out? What help of his does he think you are you refusing?
The fact that you're talking about it means that you ARE trying and he must know that you're feeling distressed about it.
Would he be prepared to go to counselling with you?

