My wife told me that she has a huge crush on my friend, he and his wife are swingers.

My wife told me about the crush after a verry close encounter with my friend and his wife. It didn't go well. I got scaired and stopped. A few weeks later, after the close encounter, my wife said (with tear in her eyes)"i hope that he doesn't hate me". A week or so later we got together for a christmas party a another friends house, his wife was hitting on me hard and my wife was flirting with him in front of all of my other friends. As the night went on I told my friends wife that I had no interest, and shortly after she talked her husband into taking her home, as they were leaving, he said in stern comment to the group "i'll be back for her", when he got back we were talking about getting a cab ride home, he got verry upset and tried to pick a fight with me. My was scared quiet becouse I would have hurt him badly(i've got skills)and would have been arrested. It took me a long time, with promting, to talk to him. Now my wife misses partying with them and is pushing me into partying with them again. She told me that nothing will happen, but I just don't know if I can trust him after he told me that he has wanted my wife ever sinse he met her, and she told me that she is verry attracted to him. She told me that on more than one occasion he has said"i should take you to the back room anyway". I've told my wife several times that it is not going to happen, and she seems to understand how I feel. As of late I've been having trouble in bed. I know the problem is stress about the situation, but how do I get past it.