Thanks to all - I think I somehow gave the impression that I am thinking about acting - by talking about it at least -which isn't true. I just fantasize about it... but in that fantasy, we both confess our feelings and then mutually decide that we should never do anything about it because we love our spouses & kids. I don't fantasize about an affair. I think the line of the fantasy is drawn much more innocently than that!
I appreciate all your thoughts - all good ideas. I think that I need to deliberately focus more on my husband when I am with the other couple. I know there is attraction between my husband and the wife of the couple as well. I think it is all okay as long as (as one poster said) we never spend any time with the opposite sex person alone.