Sorry to be so paranoid, but how do you know that he isn't having people over to do drugs or to have orgies or engage in child pornography? From how he sounds, he obviously has no regard for the rights of others. Not all sociopaths kill people. Others take advantage of them simply because they can.
Even if he isn't this bad, he is clearly taking advantage of you because he knows he can. The people who claim to not know that they are either hurting you or taking advantage of you ALWAYS have some clue of it. He's also using the "they let me use it, so I can do what I want" excuse to defraud you. Forget about the family connection to this person. If he wasn't a relative, you would have no problem getting the sheriff's department to have him removed when you showed up and saw him there.
As far as your husband goes, he needs to wake up and smell the potential for disaster. Say something does happen. That nitwit nephew won't be responsible. Whoever is on the deed and mortgage will. Is your financial future so secure that you can overcome a lawsuit, property damage, an increase in insurance premiums, etc?
As for the rest of your freeloading family, tell them that your greedy nephew abused his privilege and that you were trying to help him out in what you thought was a time of need. Your vacation house is not up for grabs. There are too many issues of liability to consider. Anyone with half a brain and any sense of decency would never ask to stay there or ask for their own key anyway. Sorry to be so crude, but I have to wonder about the integrity of these people. Where do they get this sense of entitlement?
The first step I would take is to have the locks changed. I would notify your nephew in writing of this. Forget about damaging the family relationship because it sounds like things are incredibly toxic anyway, but no one likes to talk about it. Sure standing up for yourselves may cause a problem, but are these awful people really worth your time? You could pick people off the street to treat you better.
Tell anyone else who asks about the place that it's not up for grabs. You would be more than happy to invite people along when you are going, but allowing people into the house when you are not there is a liability issue. If they don't like it, that's one less Christmas card you have to send.
Life is too short to have to deal with toxic people. Especially relatives. And tell that husband of yours that you are trying to protect your financial security. If he's like every other man, he will appreciate it.