When I was younger my parents went on vacation with us and sometimes they went alone. I use to want to go but I got over it. Sometimes you need to some enjoyment with just the two of you.
Me and my fianc? Have a couple cruise coming up for 5 days that we both are looking forward to. My 8 year old daughter wants to come but I already explained she can't and she understand that. She might not want to accept it but she will.
So it is okay if the two of you go on one trip out of the year without your son. I see no harm with doing so.