Should he have been invited, yes, and it will only cause more by him knowing you did not invite him

But you are correct, it was YOUR party, you were the host, and it was for husband. If someone...