I have gone out with this guy(he is 47 and so am I) now 3 times. He has told me he is into, swinging,group sex and watching his girlfriend having sex with others.(everything else about him is great, but that) I told him I am not into that sort of sex, just one on one (monagamy). I feel I should not see him anymore. My gf's say I should have fun with him, but I can't. I am new to dating but would like to know is that how most men out there are today and I am right in moving on?

You should go with your instincts. If you get pressured into something that makes you uncomfortable, it would likely be an unpleasant experience.
I'm not into group sex. And I personally know none of my friends are either.